[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:42:22 dist-test-slave-mld5 krb5kdc[22084](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:42:22 dist-test-slave-mld5 krb5kdc[22084](info): set up 2 sockets
Jun 19 18:42:22 dist-test-slave-mld5 krb5kdc[22084](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 18:42:23.620030 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:42:23 dist-test-slave-mld5 krb5kdc[22084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:42:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:42:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.143.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
I20260619 18:42:23.738312 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:32825
--webserver_interface=127.21.143.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:38311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:32825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:23.846709 22097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:23.846971 22097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:23.847010 22097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:23.851030 22097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:42:23.851085 22097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:23.851119 22097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:23.851173 22097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:42:23.851207 22097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:42:23.856351 22097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:38311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:32825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:32825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:23.857340 22097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:23.858337 22097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:23.866567 22102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:23.866582 22097 server_base.cc:1034] running on GCE node
W20260619 18:42:23.866571 22105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:23.866600 22103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:23.867375 22097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:23.868470 22097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:23.869657 22097 hybrid_clock.cc:648] HybridClock initialized: now 1781894543869644 us; error 26 us; skew 500 ppm
Jun 19 18:42:23 dist-test-slave-mld5 krb5kdc[22084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894543, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:23.872361 22097 init.cc:377] Logged in from keytab as kudu/127.21.143.126@KRBTEST.COM (short username kudu)
I20260619 18:42:23.873252 22097 webserver.cc:460] Webserver started at http://127.21.143.126:39899/ using document root <none> and password file <none>
I20260619 18:42:23.873811 22097 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:23.873901 22097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:23.874128 22097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:23.876297 22097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "3998ab01e7d14730a6acf1b0aced5465"
format_stamp: "Formatted at 2026-06-19 18:42:23 on dist-test-slave-mld5"
server_key: "48fc8d28cc1376ca34f889f93df55f95"
server_key_iv: "25bb4e479bc5642e92465af8b2d3c83c"
server_key_version: "encryptionkey@0"
I20260619 18:42:23.876900 22097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "3998ab01e7d14730a6acf1b0aced5465"
format_stamp: "Formatted at 2026-06-19 18:42:23 on dist-test-slave-mld5"
server_key: "48fc8d28cc1376ca34f889f93df55f95"
server_key_iv: "25bb4e479bc5642e92465af8b2d3c83c"
server_key_version: "encryptionkey@0"
I20260619 18:42:23.880684 22097 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:42:23.883129 22112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:23.884552 22097 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:42:23.884698 22097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "3998ab01e7d14730a6acf1b0aced5465"
format_stamp: "Formatted at 2026-06-19 18:42:23 on dist-test-slave-mld5"
server_key: "48fc8d28cc1376ca34f889f93df55f95"
server_key_iv: "25bb4e479bc5642e92465af8b2d3c83c"
server_key_version: "encryptionkey@0"
I20260619 18:42:23.884819 22097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:23.913177 22097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:23.916296 22097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:23.916512 22097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:23.923926 22097 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:32825
I20260619 18:42:23.923951 22164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:32825 every 8 connection(s)
I20260619 18:42:23.924896 22097 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:42:23.927670 22165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:23.933082 22165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3998ab01e7d14730a6acf1b0aced5465: Bootstrap starting.
I20260619 18:42:23.934903 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 22097
I20260619 18:42:23.935077 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:42:23.935884 22165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3998ab01e7d14730a6acf1b0aced5465: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:23.936721 22077 external_mini_cluster.cc:1428] Setting key 62d6a702e6395ce01ed2a3d317df75bf
I20260619 18:42:23.936736 22165 log.cc:824] T 00000000000000000000000000000000 P 3998ab01e7d14730a6acf1b0aced5465: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:23.938884 22165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3998ab01e7d14730a6acf1b0aced5465: No bootstrap required, opened a new log
I20260619 18:42:23.942878 22165 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3998ab01e7d14730a6acf1b0aced5465 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3998ab01e7d14730a6acf1b0aced5465" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 32825 } }
I20260619 18:42:23.943092 22165 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3998ab01e7d14730a6acf1b0aced5465 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:23.943126 22165 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3998ab01e7d14730a6acf1b0aced5465 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3998ab01e7d14730a6acf1b0aced5465, State: Initialized, Role: FOLLOWER
I20260619 18:42:23.943634 22165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3998ab01e7d14730a6acf1b0aced5465 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3998ab01e7d14730a6acf1b0aced5465" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 32825 } }
I20260619 18:42:23.943758 22165 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3998ab01e7d14730a6acf1b0aced5465 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:42:23.943845 22165 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3998ab01e7d14730a6acf1b0aced5465 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:42:23.943980 22165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3998ab01e7d14730a6acf1b0aced5465 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:23.945081 22165 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3998ab01e7d14730a6acf1b0aced5465 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3998ab01e7d14730a6acf1b0aced5465" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 32825 } }
I20260619 18:42:23.945443 22165 leader_election.cc:302] T 00000000000000000000000000000000 P 3998ab01e7d14730a6acf1b0aced5465 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3998ab01e7d14730a6acf1b0aced5465; no voters:
I20260619 18:42:23.945776 22165 leader_election.cc:288] T 00000000000000000000000000000000 P 3998ab01e7d14730a6acf1b0aced5465 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:42:23.945953 22170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3998ab01e7d14730a6acf1b0aced5465 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:23.946249 22170 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3998ab01e7d14730a6acf1b0aced5465 [term 1 LEADER]: Becoming Leader. State: Replica: 3998ab01e7d14730a6acf1b0aced5465, State: Running, Role: LEADER
I20260619 18:42:23.946614 22170 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3998ab01e7d14730a6acf1b0aced5465 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3998ab01e7d14730a6acf1b0aced5465" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 32825 } }
I20260619 18:42:23.947013 22165 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3998ab01e7d14730a6acf1b0aced5465 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:42:23.948321 22172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3998ab01e7d14730a6acf1b0aced5465 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3998ab01e7d14730a6acf1b0aced5465. Latest consensus state: current_term: 1 leader_uuid: "3998ab01e7d14730a6acf1b0aced5465" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3998ab01e7d14730a6acf1b0aced5465" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 32825 } } }
I20260619 18:42:23.948439 22172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3998ab01e7d14730a6acf1b0aced5465 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:23.948776 22171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3998ab01e7d14730a6acf1b0aced5465 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3998ab01e7d14730a6acf1b0aced5465" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3998ab01e7d14730a6acf1b0aced5465" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 32825 } } }
I20260619 18:42:23.948864 22171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3998ab01e7d14730a6acf1b0aced5465 [sys.catalog]: This master's current role is: LEADER
W20260619 18:42:23.951754 22185 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3998ab01e7d14730a6acf1b0aced5465: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:42:23.951825 22185 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:42:23.951879 22180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:42:23.952715 22180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 18:42:23 dist-test-slave-mld5 krb5kdc[22084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:23.957633 22180 catalog_manager.cc:1347] Generated new cluster ID: d8f63c83010744188de8f655912805eb
I20260619 18:42:23.957685 22180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:42:23.964121 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:23.940033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34586 (local address 127.21.143.126:32825)
0619 18:42:23.940473 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:23.940494 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:23.941674 (+ 1180us) server_negotiation.cc:407] Connection header received
0619 18:42:23.944637 (+ 2963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:23.944656 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:23.945068 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:23.945433 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:23.948414 (+ 2981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:23.949937 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:23.951674 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:23.952381 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:23.952546 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:23.957465 (+ 4919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:23.957472 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:23.957487 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:23.957519 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:23.960429 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:23.961029 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:23.961035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:23.961041 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:23.961180 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:23.961543 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:23.961546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:23.961548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:23.962069 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:23.962279 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:23.962864 (+ 585us) server_negotiation.cc:299] Negotiation successful
0619 18:42:23.963159 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":104,"threads_started":1}
I20260619 18:42:23.969024 22180 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:42:23.970105 22180 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:42:23.976884 22180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3998ab01e7d14730a6acf1b0aced5465: Generated new TSK 0
I20260619 18:42:23.977648 22180 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.143.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab.
Entry for principal kudu/127.21.143.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab.
I20260619 18:42:24.008370 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.65:0
--local_ip_for_outbound_sockets=127.21.143.65
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:32825
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:38311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:24.124532 22191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:24.124815 22191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:24.124892 22191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:24.129091 22191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:24.129186 22191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:24.129279 22191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.65
I20260619 18:42:24.134384 22191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:38311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:32825
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.65
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:24.135478 22191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:24.136477 22191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 18:42:24.145825 22191 server_base.cc:1034] running on GCE node
W20260619 18:42:24.145846 22197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:24.145843 22196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:24.145864 22199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:24.146417 22191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:24.147003 22191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:24.148195 22191 hybrid_clock.cc:648] HybridClock initialized: now 1781894544148154 us; error 54 us; skew 500 ppm
Jun 19 18:42:24 dist-test-slave-mld5 krb5kdc[22084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894544, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:24.150972 22191 init.cc:377] Logged in from keytab as kudu/127.21.143.65@KRBTEST.COM (short username kudu)
I20260619 18:42:24.151895 22191 webserver.cc:460] Webserver started at http://127.21.143.65:46495/ using document root <none> and password file <none>
I20260619 18:42:24.152462 22191 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:24.152562 22191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:24.152799 22191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:24.155020 22191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-0/data/instance:
uuid: "aac09ebcd930453481371348793ac930"
format_stamp: "Formatted at 2026-06-19 18:42:24 on dist-test-slave-mld5"
server_key: "5f8f8ddf9daec87f648f2687e3ef8b4d"
server_key_iv: "1b12959b4275d1f47314a9a6038e8128"
server_key_version: "encryptionkey@0"
I20260619 18:42:24.155637 22191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance:
uuid: "aac09ebcd930453481371348793ac930"
format_stamp: "Formatted at 2026-06-19 18:42:24 on dist-test-slave-mld5"
server_key: "5f8f8ddf9daec87f648f2687e3ef8b4d"
server_key_iv: "1b12959b4275d1f47314a9a6038e8128"
server_key_version: "encryptionkey@0"
I20260619 18:42:24.159503 22191 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 18:42:24.161881 22206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:24.163066 22191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:42:24.163197 22191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-0/wal
uuid: "aac09ebcd930453481371348793ac930"
format_stamp: "Formatted at 2026-06-19 18:42:24 on dist-test-slave-mld5"
server_key: "5f8f8ddf9daec87f648f2687e3ef8b4d"
server_key_iv: "1b12959b4275d1f47314a9a6038e8128"
server_key_version: "encryptionkey@0"
I20260619 18:42:24.163342 22191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:24.184549 22191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:24.187461 22191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:24.187721 22191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:24.188319 22191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:24.189193 22191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:24.189262 22191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:24.189333 22191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:24.189384 22191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:24.198282 22191 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.65:35803
I20260619 18:42:24.198305 22319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.65:35803 every 8 connection(s)
I20260619 18:42:24.199167 22191 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
I20260619 18:42:24.205102 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 22191
I20260619 18:42:24.205214 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance
I20260619 18:42:24.205461 22077 external_mini_cluster.cc:1428] Setting key 75a5a7f5b784e2554ea50cadc9c5a167
Jun 19 18:42:24 dist-test-slave-mld5 krb5kdc[22084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894544, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:24.216302 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:24.201077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:35881 (local address 127.21.143.126:32825)
0619 18:42:24.201212 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:24.201216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:24.202170 (+ 954us) server_negotiation.cc:407] Connection header received
0619 18:42:24.203574 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:24.203577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:24.203623 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:24.203699 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:24.204909 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:24.205713 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:24.207490 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:24.207962 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:24.208062 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:24.211456 (+ 3394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:24.211463 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:24.211484 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:24.211516 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:24.213535 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:24.214207 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:24.214212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:24.214215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:24.214305 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:24.214712 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:24.214716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:24.214718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:24.214965 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:24.215075 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:24.215930 (+ 855us) server_negotiation.cc:299] Negotiation successful
0619 18:42:24.216093 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 18:42:24.217495 22322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:24.201547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:32825 (local address 127.21.143.65:35881)
0619 18:42:24.202017 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:24.202065 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:24.203366 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:24.203901 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:24.203909 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:24.204262 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:24.204743 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:24.204757 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:24.205837 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:24.205840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:24.207351 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:24.207364 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:24.208145 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:24.208153 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:24.208391 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:24.209122 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:24.209148 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:24.211222 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:24.213659 (+ 2437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:24.213668 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:24.213686 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:24.214077 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:24.214405 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:24.214408 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:24.214410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:24.214595 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:24.215223 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:24.215230 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:24.215600 (+ 370us) client_negotiation.cc:769] Sending connection context
0619 18:42:24.215826 (+ 226us) client_negotiation.cc:240] Negotiation successful
0619 18:42:24.216305 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":135,"threads_started":1}
WARNING: no policy specified for kudu/127.21.143.66@KRBTEST.COM; defaulting to no policy
I20260619 18:42:24.218895 22320 heartbeater.cc:347] Connected to a master server at 127.21.143.126:32825
I20260619 18:42:24.219182 22320 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:24.219789 22320 heartbeater.cc:511] Master 127.21.143.126:32825 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.66@KRBTEST.COM" created.
I20260619 18:42:24.221437 22129 ts_manager.cc:194] Registered new tserver with Master: aac09ebcd930453481371348793ac930 (127.21.143.65:35803)
I20260619 18:42:24.222967 22129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.65@KRBTEST.COM'} at 127.21.143.65:35881
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab.
Entry for principal kudu/127.21.143.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab.
I20260619 18:42:24.242519 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.66:0
--local_ip_for_outbound_sockets=127.21.143.66
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:32825
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:38311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:24.353588 22325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:24.353871 22325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:24.353914 22325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:24.357914 22325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:24.357970 22325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:24.358072 22325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.66
I20260619 18:42:24.363271 22325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:38311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:32825
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.66
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:24.364321 22325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:24.365370 22325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:24.374987 22331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:24.375015 22330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:24.375288 22325 server_base.cc:1034] running on GCE node
W20260619 18:42:24.375479 22333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:24.375895 22325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:24.376461 22325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:24.377642 22325 hybrid_clock.cc:648] HybridClock initialized: now 1781894544377607 us; error 46 us; skew 500 ppm
Jun 19 18:42:24 dist-test-slave-mld5 krb5kdc[22084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894544, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:24.380363 22325 init.cc:377] Logged in from keytab as kudu/127.21.143.66@KRBTEST.COM (short username kudu)
I20260619 18:42:24.381242 22325 webserver.cc:460] Webserver started at http://127.21.143.66:45539/ using document root <none> and password file <none>
I20260619 18:42:24.381817 22325 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:24.381875 22325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:24.382079 22325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:24.384254 22325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-1/data/instance:
uuid: "d154a20ba31240108957d98d3379bafb"
format_stamp: "Formatted at 2026-06-19 18:42:24 on dist-test-slave-mld5"
server_key: "3ea8c3af55139c1ed4c743e0fe78f012"
server_key_iv: "abd936cb212a29e4c3269d5bd07b8024"
server_key_version: "encryptionkey@0"
I20260619 18:42:24.384790 22325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance:
uuid: "d154a20ba31240108957d98d3379bafb"
format_stamp: "Formatted at 2026-06-19 18:42:24 on dist-test-slave-mld5"
server_key: "3ea8c3af55139c1ed4c743e0fe78f012"
server_key_iv: "abd936cb212a29e4c3269d5bd07b8024"
server_key_version: "encryptionkey@0"
I20260619 18:42:24.388401 22325 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 18:42:24.390702 22340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:24.392047 22325 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 18:42:24.392171 22325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-1/wal
uuid: "d154a20ba31240108957d98d3379bafb"
format_stamp: "Formatted at 2026-06-19 18:42:24 on dist-test-slave-mld5"
server_key: "3ea8c3af55139c1ed4c743e0fe78f012"
server_key_iv: "abd936cb212a29e4c3269d5bd07b8024"
server_key_version: "encryptionkey@0"
I20260619 18:42:24.392287 22325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:24.410820 22325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:24.413851 22325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:24.414060 22325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:24.414671 22325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:24.415570 22325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:24.415624 22325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:24.415705 22325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:24.415736 22325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:24.424691 22325 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.66:40859
I20260619 18:42:24.424710 22453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.66:40859 every 8 connection(s)
I20260619 18:42:24.425614 22325 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
I20260619 18:42:24.428941 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 22325
I20260619 18:42:24.429083 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance
I20260619 18:42:24.429364 22077 external_mini_cluster.cc:1428] Setting key 1482e9857f39b634feed69cad452da38
Jun 19 18:42:24 dist-test-slave-mld5 krb5kdc[22084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894544, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.143.67@KRBTEST.COM; defaulting to no policy
I20260619 18:42:24.440318 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:24.427403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:58539 (local address 127.21.143.126:32825)
0619 18:42:24.427540 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:24.427544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:24.428361 (+ 817us) server_negotiation.cc:407] Connection header received
0619 18:42:24.429987 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:24.429990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:24.430040 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:24.430110 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:24.431664 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:24.432198 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:24.433472 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:24.433801 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:24.433928 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:24.436643 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:24.436646 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:24.436648 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:24.436671 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:24.438128 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:24.438632 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:24.438640 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:24.438641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:24.438711 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:24.439085 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:24.439088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:24.439090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:24.439264 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:24.439413 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:24.439968 (+ 555us) server_negotiation.cc:299] Negotiation successful
0619 18:42:24.440094 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 18:42:24.441223 22456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:24.427756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:32825 (local address 127.21.143.66:58539)
0619 18:42:24.428198 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:24.428241 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:24.429746 (+ 1505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:24.430312 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:24.430320 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:24.430778 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:24.431506 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:24.431529 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:24.432327 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:24.432333 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:24.433331 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:24.433340 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:24.433962 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:24.433967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:24.434143 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:24.434775 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:24.434791 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:24.436469 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:24.438245 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:24.438251 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:24.438266 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:24.438519 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:24.438810 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:24.438813 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:24.438815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:24.438980 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:24.439422 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:24.439430 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:24.439704 (+ 274us) client_negotiation.cc:769] Sending connection context
0619 18:42:24.439912 (+ 208us) client_negotiation.cc:240] Negotiation successful
0619 18:42:24.440264 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":122,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.67@KRBTEST.COM" created.
I20260619 18:42:24.442464 22454 heartbeater.cc:347] Connected to a master server at 127.21.143.126:32825
I20260619 18:42:24.442727 22454 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:24.443352 22454 heartbeater.cc:511] Master 127.21.143.126:32825 requested a full tablet report, sending...
I20260619 18:42:24.444422 22129 ts_manager.cc:194] Registered new tserver with Master: d154a20ba31240108957d98d3379bafb (127.21.143.66:40859)
I20260619 18:42:24.445004 22129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.66@KRBTEST.COM'} at 127.21.143.66:58539
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab.
Entry for principal kudu/127.21.143.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab.
I20260619 18:42:24.457859 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.67:0
--local_ip_for_outbound_sockets=127.21.143.67
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:32825
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:38311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:24.569128 22459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:24.569398 22459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:24.569475 22459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:24.573753 22459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:24.573808 22459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:24.573913 22459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.67
I20260619 18:42:24.579049 22459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:38311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:32825
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.67
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:24.580120 22459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:24.581192 22459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:24.590975 22465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:24.591150 22459 server_base.cc:1034] running on GCE node
W20260619 18:42:24.590969 22464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:24.591051 22467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:24.591759 22459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:24.592372 22459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:24.593532 22459 hybrid_clock.cc:648] HybridClock initialized: now 1781894544593498 us; error 52 us; skew 500 ppm
Jun 19 18:42:24 dist-test-slave-mld5 krb5kdc[22084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894544, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:24.596596 22459 init.cc:377] Logged in from keytab as kudu/127.21.143.67@KRBTEST.COM (short username kudu)
I20260619 18:42:24.597470 22459 webserver.cc:460] Webserver started at http://127.21.143.67:45751/ using document root <none> and password file <none>
I20260619 18:42:24.597990 22459 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:24.598047 22459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:24.598212 22459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:24.600255 22459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-2/data/instance:
uuid: "5f56ee6adf9d487db59dbece2d97e35f"
format_stamp: "Formatted at 2026-06-19 18:42:24 on dist-test-slave-mld5"
server_key: "d27ea74f93c043ac66939c45ae5a4ee1"
server_key_iv: "e859c6b64ee84cfebf2dc4ace71ec427"
server_key_version: "encryptionkey@0"
I20260619 18:42:24.600752 22459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance:
uuid: "5f56ee6adf9d487db59dbece2d97e35f"
format_stamp: "Formatted at 2026-06-19 18:42:24 on dist-test-slave-mld5"
server_key: "d27ea74f93c043ac66939c45ae5a4ee1"
server_key_iv: "e859c6b64ee84cfebf2dc4ace71ec427"
server_key_version: "encryptionkey@0"
I20260619 18:42:24.604413 22459 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:42:24.607127 22474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:24.608368 22459 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:42:24.608494 22459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-2/wal
uuid: "5f56ee6adf9d487db59dbece2d97e35f"
format_stamp: "Formatted at 2026-06-19 18:42:24 on dist-test-slave-mld5"
server_key: "d27ea74f93c043ac66939c45ae5a4ee1"
server_key_iv: "e859c6b64ee84cfebf2dc4ace71ec427"
server_key_version: "encryptionkey@0"
I20260619 18:42:24.608616 22459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:24.634747 22459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:24.638302 22459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:24.638520 22459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:24.639333 22459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:24.640374 22459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:24.640448 22459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:24.640515 22459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:24.640547 22459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:24.649782 22459 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.67:45585
I20260619 18:42:24.649799 22587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.67:45585 every 8 connection(s)
I20260619 18:42:24.650748 22459 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
I20260619 18:42:24.653987 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 22459
I20260619 18:42:24.654072 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance
I20260619 18:42:24.654333 22077 external_mini_cluster.cc:1428] Setting key f8548d65b9ea69864cb9b66f847064cb
Jun 19 18:42:24 dist-test-slave-mld5 krb5kdc[22084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894544, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:24.664212 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:24.652595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:41079 (local address 127.21.143.126:32825)
0619 18:42:24.652740 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:24.652743 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:24.653575 (+ 832us) server_negotiation.cc:407] Connection header received
0619 18:42:24.655034 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:24.655038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:24.655097 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:24.655197 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:24.656424 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:24.657083 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:24.658222 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:24.658444 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:24.658549 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:24.660819 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:24.660822 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:24.660824 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:24.660845 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:24.662177 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:24.662661 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:24.662665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:24.662667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:24.662751 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:24.663085 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:24.663088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:24.663089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:24.663263 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:24.663434 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:24.663894 (+ 460us) server_negotiation.cc:299] Negotiation successful
0619 18:42:24.664044 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 18:42:24.664932 22590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:24.652996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:32825 (local address 127.21.143.67:41079)
0619 18:42:24.653425 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:24.653467 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:24.654804 (+ 1337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:24.655402 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:24.655411 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:24.655816 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:24.656284 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:24.656297 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:24.657216 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:24.657219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:24.658099 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:24.658108 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:24.658584 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:24.658588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:24.658753 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:24.659275 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:24.659290 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:24.660658 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:24.662291 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:24.662296 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:24.662309 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:24.662559 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:24.662849 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:24.662852 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:24.662853 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:24.662983 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:24.663431 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:24.663437 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:24.663663 (+ 226us) client_negotiation.cc:769] Sending connection context
0619 18:42:24.663840 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 18:42:24.664173 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":118,"threads_started":1}
I20260619 18:42:24.665911 22588 heartbeater.cc:347] Connected to a master server at 127.21.143.126:32825
I20260619 18:42:24.666147 22588 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:24.666589 22588 heartbeater.cc:511] Master 127.21.143.126:32825 requested a full tablet report, sending...
I20260619 18:42:24.667747 22129 ts_manager.cc:194] Registered new tserver with Master: 5f56ee6adf9d487db59dbece2d97e35f (127.21.143.67:45585)
I20260619 18:42:24.668445 22129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.67@KRBTEST.COM'} at 127.21.143.67:41079
I20260619 18:42:24.668568 22077 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 18:42:24 dist-test-slave-mld5 krb5kdc[22084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894544, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 18:42:24 dist-test-slave-mld5 krb5kdc[22084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894544, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.143.65@KRBTEST.COM
I20260619 18:42:24.694693 22595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:24.685613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57668 (local address 127.21.143.65:35803)
0619 18:42:24.685925 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:24.685938 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:24.685963 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:42:24.686093 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:24.686099 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:24.686297 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:24.686421 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:24.687039 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:24.687901 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:24.688718 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:24.689119 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:24.689216 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:24.691667 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:24.691674 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:24.691687 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:24.691707 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:24.693099 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:24.693469 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:24.693474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:24.693475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:24.693543 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:24.693755 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:24.693757 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:24.693758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:24.693959 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:24.694128 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:24.694329 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 18:42:24.694485 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":100,"threads_started":1}
W20260619 18:42:24.695816 22234 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57668
Jun 19 18:42:24 dist-test-slave-mld5 krb5kdc[22084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 18:42:24 dist-test-slave-mld5 krb5kdc[22084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.65@KRBTEST.COM
I20260619 18:42:24.721413 22595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:24.713123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57670 (local address 127.21.143.65:35803)
0619 18:42:24.713285 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:24.713289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:24.713427 (+ 138us) server_negotiation.cc:407] Connection header received
0619 18:42:24.713532 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:24.713534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:24.713578 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:24.713649 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:24.714274 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:24.714786 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:24.715621 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:24.715841 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:24.715920 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:24.718363 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:24.718367 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:24.718370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:24.718401 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:24.719767 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:24.720208 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:24.720212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:24.720214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:24.720302 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:24.720590 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:24.720593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:24.720595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:24.720763 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:24.720895 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:24.721132 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 18:42:24.721250 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 18:42:24.722364 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 22191
I20260619 18:42:24.728346 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 22325
I20260619 18:42:24.735719 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 22459
I20260619 18:42:24.741180 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 22097
2026-06-19T18:42:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2163 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:42:24 dist-test-slave-mld5 krb5kdc[22602](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:42:24 dist-test-slave-mld5 krb5kdc[22602](info): set up 2 sockets
Jun 19 18:42:24 dist-test-slave-mld5 krb5kdc[22602](info): commencing operation
krb5kdc: starting...
W20260619 18:42:25.787176 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:42:25 dist-test-slave-mld5 krb5kdc[22602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:42:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:42:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.143.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
I20260619 18:42:25.900516 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:45167
--webserver_interface=127.21.143.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:37237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:45167
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:26.007164 22615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:26.007663 22615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:26.007781 22615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:26.011847 22615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:42:26.011896 22615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:26.011916 22615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:26.011940 22615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:42:26.011981 22615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:42:26.017139 22615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:37237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:45167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:45167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:26.018096 22615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:26.019084 22615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:26.027678 22621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:26.027680 22620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:26.027747 22615 server_base.cc:1034] running on GCE node
W20260619 18:42:26.027680 22623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:26.028524 22615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:26.029600 22615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:26.030761 22615 hybrid_clock.cc:648] HybridClock initialized: now 1781894546030746 us; error 27 us; skew 500 ppm
Jun 19 18:42:26 dist-test-slave-mld5 krb5kdc[22602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894546, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:26.033581 22615 init.cc:377] Logged in from keytab as kudu/127.21.143.126@KRBTEST.COM (short username kudu)
I20260619 18:42:26.034422 22615 webserver.cc:460] Webserver started at http://127.21.143.126:40337/ using document root <none> and password file <none>
I20260619 18:42:26.034971 22615 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:26.035058 22615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:26.035354 22615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:26.037420 22615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "6e5064d2a7774aadaabf8a17fd1d8069"
format_stamp: "Formatted at 2026-06-19 18:42:26 on dist-test-slave-mld5"
server_key: "09a7a974af32973dd27a2ae39662f216"
server_key_iv: "af9642228bc3fea85aca89f60ff994de"
server_key_version: "encryptionkey@0"
I20260619 18:42:26.037981 22615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "6e5064d2a7774aadaabf8a17fd1d8069"
format_stamp: "Formatted at 2026-06-19 18:42:26 on dist-test-slave-mld5"
server_key: "09a7a974af32973dd27a2ae39662f216"
server_key_iv: "af9642228bc3fea85aca89f60ff994de"
server_key_version: "encryptionkey@0"
I20260619 18:42:26.041644 22615 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:42:26.044023 22630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:26.045343 22615 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:42:26.045485 22615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "6e5064d2a7774aadaabf8a17fd1d8069"
format_stamp: "Formatted at 2026-06-19 18:42:26 on dist-test-slave-mld5"
server_key: "09a7a974af32973dd27a2ae39662f216"
server_key_iv: "af9642228bc3fea85aca89f60ff994de"
server_key_version: "encryptionkey@0"
I20260619 18:42:26.045596 22615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:26.059701 22615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:26.062992 22615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:26.063185 22615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:26.070168 22615 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:45167
I20260619 18:42:26.070233 22682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:45167 every 8 connection(s)
I20260619 18:42:26.071189 22615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:42:26.074072 22683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:26.075954 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 22615
I20260619 18:42:26.076045 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:42:26.076354 22077 external_mini_cluster.cc:1428] Setting key 238d835e8518bd17f85000c9bc48d83c
I20260619 18:42:26.081001 22683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e5064d2a7774aadaabf8a17fd1d8069: Bootstrap starting.
I20260619 18:42:26.083562 22683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e5064d2a7774aadaabf8a17fd1d8069: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:26.084473 22683 log.cc:824] T 00000000000000000000000000000000 P 6e5064d2a7774aadaabf8a17fd1d8069: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:42:26 dist-test-slave-mld5 krb5kdc[22602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:26.086654 22683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e5064d2a7774aadaabf8a17fd1d8069: No bootstrap required, opened a new log
I20260619 18:42:26.091334 22683 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e5064d2a7774aadaabf8a17fd1d8069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e5064d2a7774aadaabf8a17fd1d8069" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 45167 } }
I20260619 18:42:26.091635 22683 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e5064d2a7774aadaabf8a17fd1d8069 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:26.091704 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:26.077789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50370 (local address 127.21.143.126:45167)
0619 18:42:26.078207 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:26.078227 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:26.078270 (+ 43us) server_negotiation.cc:407] Connection header received
0619 18:42:26.079112 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:26.079140 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:26.079594 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:26.079970 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:26.080574 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:26.082018 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:26.082903 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:26.083633 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:26.083812 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:26.086672 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:26.086680 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:26.086695 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:26.086725 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:26.088743 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:26.089167 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:26.089173 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:26.089179 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:26.089312 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:26.089562 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:26.089565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:26.089567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:26.090062 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:26.090290 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:26.090463 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 18:42:26.090750 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":104,"threads_started":1}
I20260619 18:42:26.091719 22683 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e5064d2a7774aadaabf8a17fd1d8069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e5064d2a7774aadaabf8a17fd1d8069, State: Initialized, Role: FOLLOWER
I20260619 18:42:26.092535 22683 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e5064d2a7774aadaabf8a17fd1d8069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e5064d2a7774aadaabf8a17fd1d8069" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 45167 } }
I20260619 18:42:26.092702 22683 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e5064d2a7774aadaabf8a17fd1d8069 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:42:26.092768 22683 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e5064d2a7774aadaabf8a17fd1d8069 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:42:26.092860 22683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e5064d2a7774aadaabf8a17fd1d8069 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:26.093667 22683 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e5064d2a7774aadaabf8a17fd1d8069 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e5064d2a7774aadaabf8a17fd1d8069" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 45167 } }
I20260619 18:42:26.094043 22683 leader_election.cc:302] T 00000000000000000000000000000000 P 6e5064d2a7774aadaabf8a17fd1d8069 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e5064d2a7774aadaabf8a17fd1d8069; no voters:
I20260619 18:42:26.094398 22683 leader_election.cc:288] T 00000000000000000000000000000000 P 6e5064d2a7774aadaabf8a17fd1d8069 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:42:26.094468 22688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e5064d2a7774aadaabf8a17fd1d8069 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:26.094722 22688 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e5064d2a7774aadaabf8a17fd1d8069 [term 1 LEADER]: Becoming Leader. State: Replica: 6e5064d2a7774aadaabf8a17fd1d8069, State: Running, Role: LEADER
I20260619 18:42:26.095037 22688 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e5064d2a7774aadaabf8a17fd1d8069 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e5064d2a7774aadaabf8a17fd1d8069" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 45167 } }
I20260619 18:42:26.095590 22683 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e5064d2a7774aadaabf8a17fd1d8069 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:42:26.096693 22689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e5064d2a7774aadaabf8a17fd1d8069 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e5064d2a7774aadaabf8a17fd1d8069" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e5064d2a7774aadaabf8a17fd1d8069" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 45167 } } }
I20260619 18:42:26.096835 22689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e5064d2a7774aadaabf8a17fd1d8069 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:26.097162 22690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e5064d2a7774aadaabf8a17fd1d8069 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e5064d2a7774aadaabf8a17fd1d8069. Latest consensus state: current_term: 1 leader_uuid: "6e5064d2a7774aadaabf8a17fd1d8069" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e5064d2a7774aadaabf8a17fd1d8069" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 45167 } } }
I20260619 18:42:26.097255 22697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:42:26.097285 22690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e5064d2a7774aadaabf8a17fd1d8069 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:26.099957 22697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:42:26.105096 22697 catalog_manager.cc:1347] Generated new cluster ID: a488663259454910a31d876ab5c9d720
I20260619 18:42:26.105159 22697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:42:26.126089 22697 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:42:26.127430 22697 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:42:26.132288 22697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e5064d2a7774aadaabf8a17fd1d8069: Generated new TSK 0
I20260619 18:42:26.133075 22697 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.143.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab.
Entry for principal kudu/127.21.143.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab.
I20260619 18:42:26.164345 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.65:0
--local_ip_for_outbound_sockets=127.21.143.65
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:45167
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:37237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:26.278913 22709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:26.279232 22709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:26.279371 22709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:26.283507 22709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:26.283610 22709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:26.283718 22709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.65
I20260619 18:42:26.289100 22709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:37237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:45167
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.65
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:26.290223 22709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:26.291290 22709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:26.300782 22717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:26.300778 22715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:26.300778 22714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:26.300854 22709 server_base.cc:1034] running on GCE node
I20260619 18:42:26.301433 22709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:26.302062 22709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:26.303246 22709 hybrid_clock.cc:648] HybridClock initialized: now 1781894546303227 us; error 38 us; skew 500 ppm
Jun 19 18:42:26 dist-test-slave-mld5 krb5kdc[22602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894546, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:26.305886 22709 init.cc:377] Logged in from keytab as kudu/127.21.143.65@KRBTEST.COM (short username kudu)
I20260619 18:42:26.306740 22709 webserver.cc:460] Webserver started at http://127.21.143.65:37783/ using document root <none> and password file <none>
I20260619 18:42:26.307278 22709 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:26.307427 22709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:26.307667 22709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:26.309690 22709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-0/data/instance:
uuid: "1ff4db92da60414980c2cfac85ef83a6"
format_stamp: "Formatted at 2026-06-19 18:42:26 on dist-test-slave-mld5"
server_key: "896237a896e07dc859aa10002fb11d5f"
server_key_iv: "21d8725c3836a1e2a7e6c1af54c999be"
server_key_version: "encryptionkey@0"
I20260619 18:42:26.310226 22709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance:
uuid: "1ff4db92da60414980c2cfac85ef83a6"
format_stamp: "Formatted at 2026-06-19 18:42:26 on dist-test-slave-mld5"
server_key: "896237a896e07dc859aa10002fb11d5f"
server_key_iv: "21d8725c3836a1e2a7e6c1af54c999be"
server_key_version: "encryptionkey@0"
I20260619 18:42:26.313720 22709 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:42:26.316097 22724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:26.317698 22709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:42:26.317830 22709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-0/wal
uuid: "1ff4db92da60414980c2cfac85ef83a6"
format_stamp: "Formatted at 2026-06-19 18:42:26 on dist-test-slave-mld5"
server_key: "896237a896e07dc859aa10002fb11d5f"
server_key_iv: "21d8725c3836a1e2a7e6c1af54c999be"
server_key_version: "encryptionkey@0"
I20260619 18:42:26.317951 22709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:26.335994 22709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:26.338778 22709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:26.338984 22709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:26.339636 22709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:26.340457 22709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:26.340503 22709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:26.340574 22709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:26.340603 22709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:26.349110 22709 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.65:35057
I20260619 18:42:26.349139 22837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.65:35057 every 8 connection(s)
I20260619 18:42:26.350057 22709 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
I20260619 18:42:26.350353 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 22709
I20260619 18:42:26.350471 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance
I20260619 18:42:26.350793 22077 external_mini_cluster.cc:1428] Setting key a3481d82bcca57e273803a2a059b3775
Jun 19 18:42:26 dist-test-slave-mld5 krb5kdc[22602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894546, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.143.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.66@KRBTEST.COM" created.
I20260619 18:42:26.363848 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:26.351863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:60185 (local address 127.21.143.126:45167)
0619 18:42:26.351982 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:26.351986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:26.352958 (+ 972us) server_negotiation.cc:407] Connection header received
0619 18:42:26.354226 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:26.354229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:26.354283 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:26.354393 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:26.355601 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:26.356126 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:26.357236 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:26.357478 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:26.357590 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:26.360111 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:26.360114 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:26.360116 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:26.360139 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:26.361705 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:26.362197 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:26.362200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:26.362201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:26.362310 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:26.362684 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:26.362689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:26.362691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:26.362882 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:26.362990 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:26.363542 (+ 552us) server_negotiation.cc:299] Negotiation successful
0619 18:42:26.363666 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 18:42:26.364708 22840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:26.352356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:45167 (local address 127.21.143.65:60185)
0619 18:42:26.352826 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:26.352868 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:26.354005 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:26.354563 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:26.354571 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:26.354947 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:26.355465 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:26.355481 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:26.356251 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:26.356259 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:26.357111 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:26.357119 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:26.357626 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:26.357630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:26.357790 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:26.358446 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:26.358462 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:26.359946 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:26.361821 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:26.361826 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:26.361837 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:26.362084 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:26.362430 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:26.362434 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:26.362436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:26.362586 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:26.362990 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:26.362998 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:26.363253 (+ 255us) client_negotiation.cc:769] Sending connection context
0619 18:42:26.363503 (+ 250us) client_negotiation.cc:240] Negotiation successful
0619 18:42:26.363827 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":127,"threads_started":1}
I20260619 18:42:26.365972 22838 heartbeater.cc:347] Connected to a master server at 127.21.143.126:45167
I20260619 18:42:26.366261 22838 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:26.366753 22838 heartbeater.cc:511] Master 127.21.143.126:45167 requested a full tablet report, sending...
I20260619 18:42:26.368263 22645 ts_manager.cc:194] Registered new tserver with Master: 1ff4db92da60414980c2cfac85ef83a6 (127.21.143.65:35057)
I20260619 18:42:26.369369 22645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.65@KRBTEST.COM'} at 127.21.143.65:60185
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab.
Entry for principal kudu/127.21.143.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab.
I20260619 18:42:26.376247 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.66:0
--local_ip_for_outbound_sockets=127.21.143.66
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:45167
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:37237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:26.485270 22843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:26.485548 22843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:26.485633 22843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:26.489604 22843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:26.489696 22843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:26.489789 22843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.66
I20260619 18:42:26.495122 22843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:37237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:45167
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.66
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:26.496202 22843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:26.497354 22843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:26.506532 22851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:26.506567 22849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:26.506532 22848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:26.506728 22843 server_base.cc:1034] running on GCE node
I20260619 18:42:26.507285 22843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:26.507902 22843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:26.509085 22843 hybrid_clock.cc:648] HybridClock initialized: now 1781894546509068 us; error 28 us; skew 500 ppm
Jun 19 18:42:26 dist-test-slave-mld5 krb5kdc[22602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894546, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:26.511843 22843 init.cc:377] Logged in from keytab as kudu/127.21.143.66@KRBTEST.COM (short username kudu)
I20260619 18:42:26.512651 22843 webserver.cc:460] Webserver started at http://127.21.143.66:45319/ using document root <none> and password file <none>
I20260619 18:42:26.513201 22843 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:26.513309 22843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:26.513532 22843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:26.515616 22843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-1/data/instance:
uuid: "7d671a5d54fa41a786c32464f691e777"
format_stamp: "Formatted at 2026-06-19 18:42:26 on dist-test-slave-mld5"
server_key: "b6966f98d0d462506fb36700e4cebf0c"
server_key_iv: "7e4b0441a15a3189c9a13793682c7612"
server_key_version: "encryptionkey@0"
I20260619 18:42:26.516173 22843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance:
uuid: "7d671a5d54fa41a786c32464f691e777"
format_stamp: "Formatted at 2026-06-19 18:42:26 on dist-test-slave-mld5"
server_key: "b6966f98d0d462506fb36700e4cebf0c"
server_key_iv: "7e4b0441a15a3189c9a13793682c7612"
server_key_version: "encryptionkey@0"
I20260619 18:42:26.519767 22843 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260619 18:42:26.522225 22858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:26.523510 22843 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:42:26.523638 22843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-1/wal
uuid: "7d671a5d54fa41a786c32464f691e777"
format_stamp: "Formatted at 2026-06-19 18:42:26 on dist-test-slave-mld5"
server_key: "b6966f98d0d462506fb36700e4cebf0c"
server_key_iv: "7e4b0441a15a3189c9a13793682c7612"
server_key_version: "encryptionkey@0"
I20260619 18:42:26.523749 22843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:26.533815 22843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:26.537201 22843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:26.537423 22843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:26.538010 22843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:26.538828 22843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:26.538897 22843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:26.538967 22843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:26.539019 22843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:26.547732 22843 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.66:35179
I20260619 18:42:26.547770 22971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.66:35179 every 8 connection(s)
I20260619 18:42:26.548578 22843 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
I20260619 18:42:26.552130 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 22843
I20260619 18:42:26.552245 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance
I20260619 18:42:26.552520 22077 external_mini_cluster.cc:1428] Setting key 9cbc45b2fafe487a45994d2acee49526
Jun 19 18:42:26 dist-test-slave-mld5 krb5kdc[22602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894546, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.143.67@KRBTEST.COM; defaulting to no policy
I20260619 18:42:26.564131 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:26.550426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:39957 (local address 127.21.143.126:45167)
0619 18:42:26.550569 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:26.550574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:26.551503 (+ 929us) server_negotiation.cc:407] Connection header received
0619 18:42:26.552633 (+ 1130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:26.552637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:26.552695 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:26.552773 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:26.554208 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:26.554714 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:26.556150 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:26.556545 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:26.556665 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:26.559629 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:26.559633 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:26.559636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:26.559682 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:26.561475 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:26.562176 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:26.562179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:26.562182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:26.562293 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:26.562771 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:26.562776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:26.562780 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:26.563046 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:26.563154 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:26.563843 (+ 689us) server_negotiation.cc:299] Negotiation successful
0619 18:42:26.563954 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 18:42:26.565254 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:26.550867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:45167 (local address 127.21.143.66:39957)
0619 18:42:26.551345 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:26.551405 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:26.552389 (+ 984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:26.552978 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:26.552985 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:26.553446 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:26.554069 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:26.554088 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:26.554834 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:26.554841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:26.556036 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:26.556046 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:26.556730 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:26.556736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:26.556962 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:26.557541 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:26.557565 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:26.559421 (+ 1856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:26.561634 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:26.561642 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:26.561658 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:26.561998 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:26.562394 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:26.562397 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:26.562399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:26.562601 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:26.563168 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:26.563174 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:26.563558 (+ 384us) client_negotiation.cc:769] Sending connection context
0619 18:42:26.563776 (+ 218us) client_negotiation.cc:240] Negotiation successful
0619 18:42:26.564108 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":102,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.67@KRBTEST.COM" created.
I20260619 18:42:26.566497 22972 heartbeater.cc:347] Connected to a master server at 127.21.143.126:45167
I20260619 18:42:26.566742 22972 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:26.567281 22972 heartbeater.cc:511] Master 127.21.143.126:45167 requested a full tablet report, sending...
I20260619 18:42:26.568642 22645 ts_manager.cc:194] Registered new tserver with Master: 7d671a5d54fa41a786c32464f691e777 (127.21.143.66:35179)
I20260619 18:42:26.569495 22645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.66@KRBTEST.COM'} at 127.21.143.66:39957
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab.
Entry for principal kudu/127.21.143.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab.
I20260619 18:42:26.581694 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.67:0
--local_ip_for_outbound_sockets=127.21.143.67
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:45167
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:37237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:26.692670 22977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:26.692960 22977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:26.693035 22977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:26.697286 22977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:26.697368 22977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:26.697508 22977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.67
I20260619 18:42:26.702844 22977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:37237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:45167
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.67
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:26.704028 22977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:26.705097 22977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:26.714679 22982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:26.714711 22985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:26.714825 22977 server_base.cc:1034] running on GCE node
W20260619 18:42:26.714988 22983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:26.715452 22977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:26.716100 22977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:26.717273 22977 hybrid_clock.cc:648] HybridClock initialized: now 1781894546717259 us; error 32 us; skew 500 ppm
Jun 19 18:42:26 dist-test-slave-mld5 krb5kdc[22602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894546, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:26.719949 22977 init.cc:377] Logged in from keytab as kudu/127.21.143.67@KRBTEST.COM (short username kudu)
I20260619 18:42:26.720709 22977 webserver.cc:460] Webserver started at http://127.21.143.67:43661/ using document root <none> and password file <none>
I20260619 18:42:26.721220 22977 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:26.721273 22977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:26.721429 22977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:26.723603 22977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-2/data/instance:
uuid: "e1191936c63c4d3f9cacd0fd2cd8819e"
format_stamp: "Formatted at 2026-06-19 18:42:26 on dist-test-slave-mld5"
server_key: "6a858c6bde3c92c840a5511cc6ab97de"
server_key_iv: "27f6e5f96dbc490a0c81a2c852671c36"
server_key_version: "encryptionkey@0"
I20260619 18:42:26.724119 22977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance:
uuid: "e1191936c63c4d3f9cacd0fd2cd8819e"
format_stamp: "Formatted at 2026-06-19 18:42:26 on dist-test-slave-mld5"
server_key: "6a858c6bde3c92c840a5511cc6ab97de"
server_key_iv: "27f6e5f96dbc490a0c81a2c852671c36"
server_key_version: "encryptionkey@0"
I20260619 18:42:26.728005 22977 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260619 18:42:26.730296 22992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:26.731541 22977 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260619 18:42:26.731631 22977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-2/wal
uuid: "e1191936c63c4d3f9cacd0fd2cd8819e"
format_stamp: "Formatted at 2026-06-19 18:42:26 on dist-test-slave-mld5"
server_key: "6a858c6bde3c92c840a5511cc6ab97de"
server_key_iv: "27f6e5f96dbc490a0c81a2c852671c36"
server_key_version: "encryptionkey@0"
I20260619 18:42:26.731711 22977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:26.744916 22977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:26.747639 22977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:26.747776 22977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:26.748278 22977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:26.749094 22977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:26.749137 22977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:26.749212 22977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:26.749249 22977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:26.757674 22977 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.67:37737
I20260619 18:42:26.757702 23105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.67:37737 every 8 connection(s)
I20260619 18:42:26.758514 22977 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
Jun 19 18:42:26 dist-test-slave-mld5 krb5kdc[22602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894546, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:26.767598 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 22977
I20260619 18:42:26.767717 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance
I20260619 18:42:26.768000 22077 external_mini_cluster.cc:1428] Setting key 40afa641f416b8e26a8f7b36ec81bdf4
I20260619 18:42:26.771823 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:26.760215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:58221 (local address 127.21.143.126:45167)
0619 18:42:26.760338 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:26.760341 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:26.761042 (+ 701us) server_negotiation.cc:407] Connection header received
0619 18:42:26.762274 (+ 1232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:26.762278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:26.762335 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:26.762449 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:26.763657 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:26.764168 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:26.765299 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:26.765534 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:26.765643 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:26.768013 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:26.768020 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:26.768022 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:26.768051 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:26.769687 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:26.770271 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:26.770276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:26.770278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:26.770361 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:26.770706 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:26.770711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:26.770713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:26.770903 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:26.771013 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:26.771550 (+ 537us) server_negotiation.cc:299] Negotiation successful
0619 18:42:26.771669 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 18:42:26.772616 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:26.760510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:45167 (local address 127.21.143.67:58221)
0619 18:42:26.760906 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:26.760946 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:26.762069 (+ 1123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:26.762623 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:26.762630 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:26.762966 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:26.763505 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:26.763522 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:26.764328 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:26.764335 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:26.765174 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:26.765182 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:26.765679 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:26.765684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:26.765842 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:26.766374 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:26.766388 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:26.767858 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:26.769794 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:26.769799 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:26.769813 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:26.770148 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:26.770454 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:26.770459 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:26.770461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:26.770609 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:26.771010 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:26.771017 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:26.771255 (+ 238us) client_negotiation.cc:769] Sending connection context
0619 18:42:26.771508 (+ 253us) client_negotiation.cc:240] Negotiation successful
0619 18:42:26.771820 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":92,"threads_started":1}
I20260619 18:42:26.773638 23106 heartbeater.cc:347] Connected to a master server at 127.21.143.126:45167
I20260619 18:42:26.773878 23106 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:26.774343 23106 heartbeater.cc:511] Master 127.21.143.126:45167 requested a full tablet report, sending...
I20260619 18:42:26.775261 22645 ts_manager.cc:194] Registered new tserver with Master: e1191936c63c4d3f9cacd0fd2cd8819e (127.21.143.67:37737)
I20260619 18:42:26.775844 22645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.67@KRBTEST.COM'} at 127.21.143.67:58221
I20260619 18:42:26.781867 22077 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:42:26.791723 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:26.785077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46338 (local address 127.21.143.126:45167)
0619 18:42:26.785245 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:26.785248 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:26.785326 (+ 78us) server_negotiation.cc:407] Connection header received
0619 18:42:26.785436 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:26.785438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:26.785472 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:26.785535 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:26.786122 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:26.786592 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:26.787425 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:26.787616 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:26.787690 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:26.788946 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:26.788949 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:26.788951 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:26.788971 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:26.790341 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:26.790690 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:26.790693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:26.790694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:26.790761 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:26.790981 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:26.790983 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:26.790984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:26.791160 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:26.791252 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:26.791454 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 18:42:26.791535 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 18:42:26.796065 22645 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:42:26.799113 22645 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:42:26.812970 23117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:26.807267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:35057 (local address 127.0.0.1:33416)
0619 18:42:26.807690 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:26.807758 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:26.807886 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:26.808736 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:26.808742 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:26.808764 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:26.808890 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:26.808897 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:26.809976 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:26.809980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:26.811239 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:26.811249 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:26.812416 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:26.812420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:26.812530 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:26.812582 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 18:42:26.812695 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 18:42:26.812830 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":61,"thread_start_us":98,"threads_started":1}
I20260619 18:42:26.812978 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:26.807449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33416 (local address 127.21.143.65:35057)
0619 18:42:26.808199 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:26.808210 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:26.808233 (+ 23us) server_negotiation.cc:407] Connection header received
0619 18:42:26.808311 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:26.808316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:26.808557 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:26.808721 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:26.808993 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:26.809847 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:26.811413 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:26.812271 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:26.812414 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:26.812560 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:26.812736 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 18:42:26.812826 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":124,"threads_started":1}
I20260619 18:42:26.813455 23120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:26.807512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:37737 (local address 127.0.0.1:36642)
0619 18:42:26.808459 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:26.808475 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:26.808566 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:26.809046 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:26.809049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:26.809064 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:26.809151 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:26.809157 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:26.810155 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:26.810158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:26.811846 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:26.811856 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:26.813027 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:26.813030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:26.813125 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:26.813156 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 18:42:26.813244 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 18:42:26.813311 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":881,"mutex_wait_us":1,"thread_start_us":113,"threads_started":1}
I20260619 18:42:26.813547 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:26.807988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36642 (local address 127.21.143.67:37737)
0619 18:42:26.808618 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:26.808632 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:26.808656 (+ 24us) server_negotiation.cc:407] Connection header received
0619 18:42:26.808737 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:26.808742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:26.808918 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:26.809070 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:26.809266 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:26.810020 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:26.811983 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:26.812752 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:26.812845 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:26.812965 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:26.813288 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 18:42:26.813393 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":153,"threads_started":1}
I20260619 18:42:26.815451 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:26.808048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:35179 (local address 127.0.0.1:46732)
0619 18:42:26.809104 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:26.809119 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:26.809191 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:26.810651 (+ 1460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:26.810654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:26.810671 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:26.810752 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:26.810758 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:26.812141 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:26.812144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:26.813454 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:26.813464 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:26.815048 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:26.815052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:26.815134 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:26.815155 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 18:42:26.815233 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 18:42:26.815323 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":999,"thread_start_us":81,"threads_started":1}
I20260619 18:42:26.815935 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:26.808182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46732 (local address 127.21.143.66:35179)
0619 18:42:26.810041 (+ 1859us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:26.810051 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:26.810067 (+ 16us) server_negotiation.cc:407] Connection header received
0619 18:42:26.810159 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:26.810163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:26.810336 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:26.810491 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:26.810877 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:26.811999 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:26.814353 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:26.814920 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:26.815048 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:26.815180 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:26.815260 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 18:42:26.815373 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1774,"thread_start_us":118,"threads_started":1}
I20260619 18:42:26.816324 23040 tablet_service.cc:1467] Processing CreateTablet for tablet cd93e953b0bf4596a70ffe928639524a (DEFAULT_TABLE table=test-table [id=9a9ddcb2e8d642528a1fb163341d5e1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:42:26.816324 22772 tablet_service.cc:1467] Processing CreateTablet for tablet cd93e953b0bf4596a70ffe928639524a (DEFAULT_TABLE table=test-table [id=9a9ddcb2e8d642528a1fb163341d5e1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:42:26.817483 23040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd93e953b0bf4596a70ffe928639524a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:26.817483 22772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd93e953b0bf4596a70ffe928639524a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:26.822444 22906 tablet_service.cc:1467] Processing CreateTablet for tablet cd93e953b0bf4596a70ffe928639524a (DEFAULT_TABLE table=test-table [id=9a9ddcb2e8d642528a1fb163341d5e1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:42:26.823567 22906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd93e953b0bf4596a70ffe928639524a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:26.825774 23124 tablet_bootstrap.cc:492] T cd93e953b0bf4596a70ffe928639524a P e1191936c63c4d3f9cacd0fd2cd8819e: Bootstrap starting.
I20260619 18:42:26.826576 23123 tablet_bootstrap.cc:492] T cd93e953b0bf4596a70ffe928639524a P 1ff4db92da60414980c2cfac85ef83a6: Bootstrap starting.
I20260619 18:42:26.828104 23124 tablet_bootstrap.cc:654] T cd93e953b0bf4596a70ffe928639524a P e1191936c63c4d3f9cacd0fd2cd8819e: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:26.828821 23123 tablet_bootstrap.cc:654] T cd93e953b0bf4596a70ffe928639524a P 1ff4db92da60414980c2cfac85ef83a6: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:26.829187 23124 log.cc:824] T cd93e953b0bf4596a70ffe928639524a P e1191936c63c4d3f9cacd0fd2cd8819e: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:26.830053 23123 log.cc:824] T cd93e953b0bf4596a70ffe928639524a P 1ff4db92da60414980c2cfac85ef83a6: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:26.831195 23124 tablet_bootstrap.cc:492] T cd93e953b0bf4596a70ffe928639524a P e1191936c63c4d3f9cacd0fd2cd8819e: No bootstrap required, opened a new log
I20260619 18:42:26.831440 23124 ts_tablet_manager.cc:1397] T cd93e953b0bf4596a70ffe928639524a P e1191936c63c4d3f9cacd0fd2cd8819e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 18:42:26.831951 23127 tablet_bootstrap.cc:492] T cd93e953b0bf4596a70ffe928639524a P 7d671a5d54fa41a786c32464f691e777: Bootstrap starting.
I20260619 18:42:26.832240 23123 tablet_bootstrap.cc:492] T cd93e953b0bf4596a70ffe928639524a P 1ff4db92da60414980c2cfac85ef83a6: No bootstrap required, opened a new log
I20260619 18:42:26.832417 23123 ts_tablet_manager.cc:1397] T cd93e953b0bf4596a70ffe928639524a P 1ff4db92da60414980c2cfac85ef83a6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 18:42:26.834427 23127 tablet_bootstrap.cc:654] T cd93e953b0bf4596a70ffe928639524a P 7d671a5d54fa41a786c32464f691e777: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:26.834537 23124 raft_consensus.cc:348] T cd93e953b0bf4596a70ffe928639524a P e1191936c63c4d3f9cacd0fd2cd8819e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff4db92da60414980c2cfac85ef83a6" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 35057 } } peers { permanent_uuid: "e1191936c63c4d3f9cacd0fd2cd8819e" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37737 } } peers { permanent_uuid: "7d671a5d54fa41a786c32464f691e777" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35179 } }
I20260619 18:42:26.834812 23124 raft_consensus.cc:374] T cd93e953b0bf4596a70ffe928639524a P e1191936c63c4d3f9cacd0fd2cd8819e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:26.834884 23124 raft_consensus.cc:729] T cd93e953b0bf4596a70ffe928639524a P e1191936c63c4d3f9cacd0fd2cd8819e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1191936c63c4d3f9cacd0fd2cd8819e, State: Initialized, Role: FOLLOWER
I20260619 18:42:26.835341 23127 log.cc:824] T cd93e953b0bf4596a70ffe928639524a P 7d671a5d54fa41a786c32464f691e777: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:26.835374 23124 consensus_queue.cc:260] T cd93e953b0bf4596a70ffe928639524a P e1191936c63c4d3f9cacd0fd2cd8819e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff4db92da60414980c2cfac85ef83a6" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 35057 } } peers { permanent_uuid: "e1191936c63c4d3f9cacd0fd2cd8819e" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37737 } } peers { permanent_uuid: "7d671a5d54fa41a786c32464f691e777" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35179 } }
I20260619 18:42:26.836493 23124 ts_tablet_manager.cc:1428] T cd93e953b0bf4596a70ffe928639524a P e1191936c63c4d3f9cacd0fd2cd8819e: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 18:42:26.836416 23123 raft_consensus.cc:348] T cd93e953b0bf4596a70ffe928639524a P 1ff4db92da60414980c2cfac85ef83a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff4db92da60414980c2cfac85ef83a6" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 35057 } } peers { permanent_uuid: "e1191936c63c4d3f9cacd0fd2cd8819e" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37737 } } peers { permanent_uuid: "7d671a5d54fa41a786c32464f691e777" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35179 } }
I20260619 18:42:26.836663 23123 raft_consensus.cc:374] T cd93e953b0bf4596a70ffe928639524a P 1ff4db92da60414980c2cfac85ef83a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:26.836725 23123 raft_consensus.cc:729] T cd93e953b0bf4596a70ffe928639524a P 1ff4db92da60414980c2cfac85ef83a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ff4db92da60414980c2cfac85ef83a6, State: Initialized, Role: FOLLOWER
I20260619 18:42:26.836865 23106 heartbeater.cc:503] Master 127.21.143.126:45167 was elected leader, sending a full tablet report...
I20260619 18:42:26.837180 23123 consensus_queue.cc:260] T cd93e953b0bf4596a70ffe928639524a P 1ff4db92da60414980c2cfac85ef83a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff4db92da60414980c2cfac85ef83a6" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 35057 } } peers { permanent_uuid: "e1191936c63c4d3f9cacd0fd2cd8819e" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37737 } } peers { permanent_uuid: "7d671a5d54fa41a786c32464f691e777" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35179 } }
I20260619 18:42:26.837854 22838 heartbeater.cc:503] Master 127.21.143.126:45167 was elected leader, sending a full tablet report...
I20260619 18:42:26.838153 23123 ts_tablet_manager.cc:1428] T cd93e953b0bf4596a70ffe928639524a P 1ff4db92da60414980c2cfac85ef83a6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 18:42:26.839807 23127 tablet_bootstrap.cc:492] T cd93e953b0bf4596a70ffe928639524a P 7d671a5d54fa41a786c32464f691e777: No bootstrap required, opened a new log
I20260619 18:42:26.840006 23127 ts_tablet_manager.cc:1397] T cd93e953b0bf4596a70ffe928639524a P 7d671a5d54fa41a786c32464f691e777: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 18:42:26.843444 23127 raft_consensus.cc:348] T cd93e953b0bf4596a70ffe928639524a P 7d671a5d54fa41a786c32464f691e777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff4db92da60414980c2cfac85ef83a6" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 35057 } } peers { permanent_uuid: "e1191936c63c4d3f9cacd0fd2cd8819e" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37737 } } peers { permanent_uuid: "7d671a5d54fa41a786c32464f691e777" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35179 } }
I20260619 18:42:26.843767 23127 raft_consensus.cc:374] T cd93e953b0bf4596a70ffe928639524a P 7d671a5d54fa41a786c32464f691e777 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:26.843832 23127 raft_consensus.cc:729] T cd93e953b0bf4596a70ffe928639524a P 7d671a5d54fa41a786c32464f691e777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d671a5d54fa41a786c32464f691e777, State: Initialized, Role: FOLLOWER
I20260619 18:42:26.844249 23127 consensus_queue.cc:260] T cd93e953b0bf4596a70ffe928639524a P 7d671a5d54fa41a786c32464f691e777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff4db92da60414980c2cfac85ef83a6" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 35057 } } peers { permanent_uuid: "e1191936c63c4d3f9cacd0fd2cd8819e" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37737 } } peers { permanent_uuid: "7d671a5d54fa41a786c32464f691e777" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35179 } }
I20260619 18:42:26.844892 22972 heartbeater.cc:503] Master 127.21.143.126:45167 was elected leader, sending a full tablet report...
I20260619 18:42:26.845280 23127 ts_tablet_manager.cc:1428] T cd93e953b0bf4596a70ffe928639524a P 7d671a5d54fa41a786c32464f691e777: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260619 18:42:26.851339 22839 tablet.cc:2369] T cd93e953b0bf4596a70ffe928639524a P 1ff4db92da60414980c2cfac85ef83a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:42:26.951751 23130 raft_consensus.cc:482] T cd93e953b0bf4596a70ffe928639524a P 1ff4db92da60414980c2cfac85ef83a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:42:26.952011 23130 raft_consensus.cc:504] T cd93e953b0bf4596a70ffe928639524a P 1ff4db92da60414980c2cfac85ef83a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff4db92da60414980c2cfac85ef83a6" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 35057 } } peers { permanent_uuid: "e1191936c63c4d3f9cacd0fd2cd8819e" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37737 } } peers { permanent_uuid: "7d671a5d54fa41a786c32464f691e777" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35179 } }
I20260619 18:42:26.953230 23130 leader_election.cc:288] T cd93e953b0bf4596a70ffe928639524a P 1ff4db92da60414980c2cfac85ef83a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1191936c63c4d3f9cacd0fd2cd8819e (127.21.143.67:37737), 7d671a5d54fa41a786c32464f691e777 (127.21.143.66:35179)
I20260619 18:42:26.958849 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:26.953740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:49647 (local address 127.21.143.67:37737)
0619 18:42:26.953867 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:26.953870 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:26.954086 (+ 216us) server_negotiation.cc:407] Connection header received
0619 18:42:26.954166 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:26.954169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:26.954218 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:26.954313 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:26.954730 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:26.955545 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:26.957435 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:26.958055 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:26.958186 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:26.958238 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:26.958650 (+ 412us) server_negotiation.cc:299] Negotiation successful
0619 18:42:26.958718 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:42:26.958849 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:26.953618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:54565 (local address 127.21.143.66:35179)
0619 18:42:26.953752 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:26.953757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:26.954026 (+ 269us) server_negotiation.cc:407] Connection header received
0619 18:42:26.954166 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:26.954169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:26.954222 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:26.954315 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:26.954730 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:26.955545 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:26.957435 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:26.958061 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:26.958186 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:26.958238 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:26.958650 (+ 412us) server_negotiation.cc:299] Negotiation successful
0619 18:42:26.958718 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 18:42:26.958920 23132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:26.953527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:37737 (local address 127.21.143.65:49647)
0619 18:42:26.953997 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:26.954010 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:26.954082 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:26.954342 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:26.954346 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:26.954396 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:26.954606 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:26.954615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:26.955685 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:26.955689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:26.957300 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:26.957312 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:26.958207 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:26.958211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:26.958464 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:26.958531 (+ 67us) client_negotiation.cc:769] Sending connection context
0619 18:42:26.958615 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 18:42:26.958730 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":331,"spinlock_wait_cycles":57600,"thread_start_us":95,"threads_started":1}
I20260619 18:42:26.958918 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:26.953513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:35179 (local address 127.21.143.65:54565)
0619 18:42:26.953921 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:26.953938 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:26.954068 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:26.954342 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:26.954346 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:26.954381 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:26.954620 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:26.954625 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:26.955685 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:26.955689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:26.957300 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:26.957312 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:26.958207 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:26.958211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:26.958462 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:26.958538 (+ 76us) client_negotiation.cc:769] Sending connection context
0619 18:42:26.958615 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 18:42:26.958728 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":338,"spinlock_wait_cycles":2944,"thread_start_us":105,"threads_started":1}
I20260619 18:42:26.959602 23060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd93e953b0bf4596a70ffe928639524a" candidate_uuid: "1ff4db92da60414980c2cfac85ef83a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1191936c63c4d3f9cacd0fd2cd8819e" is_pre_election: true
I20260619 18:42:26.959789 22926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd93e953b0bf4596a70ffe928639524a" candidate_uuid: "1ff4db92da60414980c2cfac85ef83a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d671a5d54fa41a786c32464f691e777" is_pre_election: true
I20260619 18:42:26.959924 23060 raft_consensus.cc:2454] T cd93e953b0bf4596a70ffe928639524a P e1191936c63c4d3f9cacd0fd2cd8819e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ff4db92da60414980c2cfac85ef83a6 in term 0.
I20260619 18:42:26.960093 22926 raft_consensus.cc:2454] T cd93e953b0bf4596a70ffe928639524a P 7d671a5d54fa41a786c32464f691e777 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ff4db92da60414980c2cfac85ef83a6 in term 0.
I20260619 18:42:26.960461 22728 leader_election.cc:302] T cd93e953b0bf4596a70ffe928639524a P 1ff4db92da60414980c2cfac85ef83a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff4db92da60414980c2cfac85ef83a6, e1191936c63c4d3f9cacd0fd2cd8819e; no voters:
I20260619 18:42:26.960744 23130 raft_consensus.cc:2781] T cd93e953b0bf4596a70ffe928639524a P 1ff4db92da60414980c2cfac85ef83a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:42:26.960842 23130 raft_consensus.cc:482] T cd93e953b0bf4596a70ffe928639524a P 1ff4db92da60414980c2cfac85ef83a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:42:26.960901 23130 raft_consensus.cc:3037] T cd93e953b0bf4596a70ffe928639524a P 1ff4db92da60414980c2cfac85ef83a6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:26.961930 23130 raft_consensus.cc:504] T cd93e953b0bf4596a70ffe928639524a P 1ff4db92da60414980c2cfac85ef83a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff4db92da60414980c2cfac85ef83a6" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 35057 } } peers { permanent_uuid: "e1191936c63c4d3f9cacd0fd2cd8819e" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37737 } } peers { permanent_uuid: "7d671a5d54fa41a786c32464f691e777" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35179 } }
I20260619 18:42:26.962276 23130 leader_election.cc:288] T cd93e953b0bf4596a70ffe928639524a P 1ff4db92da60414980c2cfac85ef83a6 [CANDIDATE]: Term 1 election: Requested vote from peers e1191936c63c4d3f9cacd0fd2cd8819e (127.21.143.67:37737), 7d671a5d54fa41a786c32464f691e777 (127.21.143.66:35179)
I20260619 18:42:26.962669 23060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd93e953b0bf4596a70ffe928639524a" candidate_uuid: "1ff4db92da60414980c2cfac85ef83a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1191936c63c4d3f9cacd0fd2cd8819e"
I20260619 18:42:26.962690 22926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd93e953b0bf4596a70ffe928639524a" candidate_uuid: "1ff4db92da60414980c2cfac85ef83a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d671a5d54fa41a786c32464f691e777"
I20260619 18:42:26.962793 22926 raft_consensus.cc:3037] T cd93e953b0bf4596a70ffe928639524a P 7d671a5d54fa41a786c32464f691e777 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:26.962793 23060 raft_consensus.cc:3037] T cd93e953b0bf4596a70ffe928639524a P e1191936c63c4d3f9cacd0fd2cd8819e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:26.964085 23060 raft_consensus.cc:2454] T cd93e953b0bf4596a70ffe928639524a P e1191936c63c4d3f9cacd0fd2cd8819e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ff4db92da60414980c2cfac85ef83a6 in term 1.
I20260619 18:42:26.964085 22926 raft_consensus.cc:2454] T cd93e953b0bf4596a70ffe928639524a P 7d671a5d54fa41a786c32464f691e777 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ff4db92da60414980c2cfac85ef83a6 in term 1.
I20260619 18:42:26.964411 22728 leader_election.cc:302] T cd93e953b0bf4596a70ffe928639524a P 1ff4db92da60414980c2cfac85ef83a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff4db92da60414980c2cfac85ef83a6, e1191936c63c4d3f9cacd0fd2cd8819e; no voters:
I20260619 18:42:26.964593 23130 raft_consensus.cc:2781] T cd93e953b0bf4596a70ffe928639524a P 1ff4db92da60414980c2cfac85ef83a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:26.964834 23130 raft_consensus.cc:686] T cd93e953b0bf4596a70ffe928639524a P 1ff4db92da60414980c2cfac85ef83a6 [term 1 LEADER]: Becoming Leader. State: Replica: 1ff4db92da60414980c2cfac85ef83a6, State: Running, Role: LEADER
I20260619 18:42:26.965176 23130 consensus_queue.cc:237] T cd93e953b0bf4596a70ffe928639524a P 1ff4db92da60414980c2cfac85ef83a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff4db92da60414980c2cfac85ef83a6" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 35057 } } peers { permanent_uuid: "e1191936c63c4d3f9cacd0fd2cd8819e" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37737 } } peers { permanent_uuid: "7d671a5d54fa41a786c32464f691e777" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35179 } }
I20260619 18:42:26.968276 22645 catalog_manager.cc:5495] T cd93e953b0bf4596a70ffe928639524a P 1ff4db92da60414980c2cfac85ef83a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ff4db92da60414980c2cfac85ef83a6 (127.21.143.65). New cstate: current_term: 1 leader_uuid: "1ff4db92da60414980c2cfac85ef83a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff4db92da60414980c2cfac85ef83a6" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 35057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1191936c63c4d3f9cacd0fd2cd8819e" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d671a5d54fa41a786c32464f691e777" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35179 } health_report { overall_health: UNKNOWN } } }
W20260619 18:42:27.009403 23107 tablet.cc:2369] T cd93e953b0bf4596a70ffe928639524a P e1191936c63c4d3f9cacd0fd2cd8819e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 18:42:27 dist-test-slave-mld5 krb5kdc[22602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.65@KRBTEST.COM
I20260619 18:42:27.015043 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:27.003394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33426 (local address 127.21.143.65:35057)
0619 18:42:27.003532 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:27.003537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:27.003701 (+ 164us) server_negotiation.cc:407] Connection header received
0619 18:42:27.003866 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:27.003872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:27.003922 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:27.004010 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:27.004767 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:27.005269 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:27.006127 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:27.006346 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:27.006492 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:27.011735 (+ 5243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:27.011756 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:27.011771 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:27.011796 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:27.013254 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:27.013715 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:27.013728 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:27.013730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:27.013815 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:27.014107 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:27.014110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:27.014112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:27.014359 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:27.014582 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:27.014733 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 18:42:27.014852 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 18:42:27 dist-test-slave-mld5 krb5kdc[22602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894547, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 18:42:27 dist-test-slave-mld5 krb5kdc[22602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894547, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.143.65@KRBTEST.COM
I20260619 18:42:27.041795 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:27.033331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33434 (local address 127.21.143.65:35057)
0619 18:42:27.033491 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:27.033497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:27.033622 (+ 125us) server_negotiation.cc:407] Connection header received
0619 18:42:27.033737 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:27.033740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:27.033784 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:27.033855 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:27.034553 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:27.035085 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:27.036023 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:27.036248 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:27.036359 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:27.038725 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:27.038728 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:27.038730 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:27.038754 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:27.040107 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:27.040523 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:27.040529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:27.040530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:27.040611 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:27.040894 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:27.040898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:27.040899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:27.041066 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:27.041229 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:27.041502 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 18:42:27.041623 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 18:42:27.049757 22973 tablet.cc:2369] T cd93e953b0bf4596a70ffe928639524a P 7d671a5d54fa41a786c32464f691e777: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 18:42:27 dist-test-slave-mld5 krb5kdc[22602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 18:42:27 dist-test-slave-mld5 krb5kdc[22602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.65@KRBTEST.COM
I20260619 18:42:27.067111 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:27.059160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33438 (local address 127.21.143.65:35057)
0619 18:42:27.059377 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:27.059381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:27.059492 (+ 111us) server_negotiation.cc:407] Connection header received
0619 18:42:27.059606 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:27.059609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:27.059667 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:27.059738 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:27.060343 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:27.060859 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:27.061655 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:27.061856 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:27.061928 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:27.064254 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:27.064257 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:27.064259 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:27.064282 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:27.065489 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:27.065888 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:27.065892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:27.065893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:27.065992 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:27.066320 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:27.066322 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:27.066323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:27.066495 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:27.066609 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:27.066805 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 18:42:27.066897 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260619 18:42:27.068500 22752 tablet_service.cc:2908] Rejecting scan request for tablet cd93e953b0bf4596a70ffe928639524a: Uninitialized: clean time has not yet been initialized
I20260619 18:42:27.070545 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 22709
I20260619 18:42:27.080497 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 22843
I20260619 18:42:27.086099 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 22977
I20260619 18:42:27.091477 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 22615
2026-06-19T18:42:27Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2350 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:42:27 dist-test-slave-mld5 krb5kdc[23148](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:42:27 dist-test-slave-mld5 krb5kdc[23148](info): set up 2 sockets
Jun 19 18:42:27 dist-test-slave-mld5 krb5kdc[23148](info): commencing operation
krb5kdc: starting...
W20260619 18:42:28.127249 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:42:28 dist-test-slave-mld5 krb5kdc[23148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:42:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:42:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.143.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
I20260619 18:42:28.245538 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:45809
--webserver_interface=127.21.143.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:37357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:45809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:28.354161 23161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:28.354455 23161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:28.354559 23161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:28.358553 23161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:42:28.358628 23161 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 18:42:28.358661 23161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:28.358726 23161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:28.358779 23161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:42:28.358816 23161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:42:28.364008 23161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:37357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:45809
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:45809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:28.365006 23161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:28.365967 23161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:28.374020 23167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:28.374154 23161 server_base.cc:1034] running on GCE node
W20260619 18:42:28.374017 23166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:28.374017 23169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:28.374884 23161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:28.375988 23161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:28.377183 23161 hybrid_clock.cc:648] HybridClock initialized: now 1781894548377170 us; error 24 us; skew 500 ppm
Jun 19 18:42:28 dist-test-slave-mld5 krb5kdc[23148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894548, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:28.379801 23161 init.cc:377] Logged in from keytab as kudu/127.21.143.126@KRBTEST.COM (short username kudu)
I20260619 18:42:28.380609 23161 webserver.cc:460] Webserver started at http://127.21.143.126:44955/ using document root <none> and password file <none>
I20260619 18:42:28.381106 23161 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:28.381172 23161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:28.381335 23161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:28.383208 23161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "ed2f4a5d94cd421eb4f87baf37b71afa"
format_stamp: "Formatted at 2026-06-19 18:42:28 on dist-test-slave-mld5"
server_key: "5137fad8914a89872d75df5d7c5fd89d"
server_key_iv: "88d9af2a0c19f5b7fec6d21d228e6ade"
server_key_version: "encryptionkey@0"
I20260619 18:42:28.383769 23161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "ed2f4a5d94cd421eb4f87baf37b71afa"
format_stamp: "Formatted at 2026-06-19 18:42:28 on dist-test-slave-mld5"
server_key: "5137fad8914a89872d75df5d7c5fd89d"
server_key_iv: "88d9af2a0c19f5b7fec6d21d228e6ade"
server_key_version: "encryptionkey@0"
I20260619 18:42:28.387185 23161 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:42:28.389485 23176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:28.390698 23161 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:42:28.390806 23161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "ed2f4a5d94cd421eb4f87baf37b71afa"
format_stamp: "Formatted at 2026-06-19 18:42:28 on dist-test-slave-mld5"
server_key: "5137fad8914a89872d75df5d7c5fd89d"
server_key_iv: "88d9af2a0c19f5b7fec6d21d228e6ade"
server_key_version: "encryptionkey@0"
I20260619 18:42:28.390951 23161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:28.419641 23161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:28.422582 23161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:28.422794 23161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:28.430579 23161 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:45809
I20260619 18:42:28.430611 23238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:45809 every 8 connection(s)
I20260619 18:42:28.431583 23161 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:42:28.431806 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 23161
I20260619 18:42:28.431921 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:42:28.432238 22077 external_mini_cluster.cc:1428] Setting key 7b1dd0f2bb60a3ad075ff5775675f2b7
I20260619 18:42:28.434960 23239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:28.440443 23239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed2f4a5d94cd421eb4f87baf37b71afa: Bootstrap starting.
Jun 19 18:42:28 dist-test-slave-mld5 krb5kdc[23148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:28.442783 23239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed2f4a5d94cd421eb4f87baf37b71afa: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:28.443959 23239 log.cc:824] T 00000000000000000000000000000000 P ed2f4a5d94cd421eb4f87baf37b71afa: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:28.445804 23239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed2f4a5d94cd421eb4f87baf37b71afa: No bootstrap required, opened a new log
I20260619 18:42:28.446903 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:28.434627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48794 (local address 127.21.143.126:45809)
0619 18:42:28.435022 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:28.435045 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:28.435281 (+ 236us) server_negotiation.cc:407] Connection header received
0619 18:42:28.436022 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:28.436038 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:28.436369 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:28.436705 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:28.437482 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:28.438464 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:28.439348 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:28.439817 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:28.439989 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:28.442425 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:28.442430 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:28.442444 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:28.442471 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:28.444214 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:28.444658 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:28.444666 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:28.444675 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:28.444783 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:28.445069 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:28.445073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:28.445074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:28.445448 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:28.445665 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:28.445894 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 18:42:28.446211 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":126,"threads_started":1}
I20260619 18:42:28.449369 23239 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed2f4a5d94cd421eb4f87baf37b71afa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2f4a5d94cd421eb4f87baf37b71afa" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 45809 } }
I20260619 18:42:28.449609 23239 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed2f4a5d94cd421eb4f87baf37b71afa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:28.449687 23239 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed2f4a5d94cd421eb4f87baf37b71afa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed2f4a5d94cd421eb4f87baf37b71afa, State: Initialized, Role: FOLLOWER
I20260619 18:42:28.450141 23239 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed2f4a5d94cd421eb4f87baf37b71afa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2f4a5d94cd421eb4f87baf37b71afa" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 45809 } }
I20260619 18:42:28.450289 23239 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed2f4a5d94cd421eb4f87baf37b71afa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:42:28.450373 23239 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed2f4a5d94cd421eb4f87baf37b71afa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:42:28.450469 23239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed2f4a5d94cd421eb4f87baf37b71afa [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:28.451401 23239 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed2f4a5d94cd421eb4f87baf37b71afa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2f4a5d94cd421eb4f87baf37b71afa" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 45809 } }
I20260619 18:42:28.451746 23239 leader_election.cc:302] T 00000000000000000000000000000000 P ed2f4a5d94cd421eb4f87baf37b71afa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed2f4a5d94cd421eb4f87baf37b71afa; no voters:
I20260619 18:42:28.452071 23239 leader_election.cc:288] T 00000000000000000000000000000000 P ed2f4a5d94cd421eb4f87baf37b71afa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:42:28.452206 23244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed2f4a5d94cd421eb4f87baf37b71afa [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:28.452443 23244 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed2f4a5d94cd421eb4f87baf37b71afa [term 1 LEADER]: Becoming Leader. State: Replica: ed2f4a5d94cd421eb4f87baf37b71afa, State: Running, Role: LEADER
I20260619 18:42:28.452808 23244 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed2f4a5d94cd421eb4f87baf37b71afa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2f4a5d94cd421eb4f87baf37b71afa" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 45809 } }
I20260619 18:42:28.453402 23239 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed2f4a5d94cd421eb4f87baf37b71afa [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:42:28.454456 23245 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed2f4a5d94cd421eb4f87baf37b71afa [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed2f4a5d94cd421eb4f87baf37b71afa. Latest consensus state: current_term: 1 leader_uuid: "ed2f4a5d94cd421eb4f87baf37b71afa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2f4a5d94cd421eb4f87baf37b71afa" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 45809 } } }
I20260619 18:42:28.454576 23245 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed2f4a5d94cd421eb4f87baf37b71afa [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:28.454869 23246 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed2f4a5d94cd421eb4f87baf37b71afa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed2f4a5d94cd421eb4f87baf37b71afa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2f4a5d94cd421eb4f87baf37b71afa" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 45809 } } }
I20260619 18:42:28.454944 23246 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed2f4a5d94cd421eb4f87baf37b71afa [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:28.454975 23253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:42:28.457998 23253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:42:28.463541 23253 catalog_manager.cc:1347] Generated new cluster ID: adb57f0271fb4dce9c1815741843aaed
I20260619 18:42:28.463614 23253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:42:28.470625 23253 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:42:28.471725 23253 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:42:28.478864 23253 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed2f4a5d94cd421eb4f87baf37b71afa: Generated new TSK 0
I20260619 18:42:28.479583 23253 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.143.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab.
Entry for principal kudu/127.21.143.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab.
I20260619 18:42:28.515061 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.65:0
--local_ip_for_outbound_sockets=127.21.143.65
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:45809
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:37357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:28.660936 23265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:28.661191 23265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:28.661234 23265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:28.665239 23265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:28.665294 23265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:28.665366 23265 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:42:28.665457 23265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.65
I20260619 18:42:28.670591 23265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:37357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.143.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.143.126:45809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.65
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:28.671729 23265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:28.672806 23265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:28.682497 23273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:28.682499 23270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:28.682499 23271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:28.682857 23265 server_base.cc:1034] running on GCE node
I20260619 18:42:28.683282 23265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:28.683933 23265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:28.685119 23265 hybrid_clock.cc:648] HybridClock initialized: now 1781894548685101 us; error 28 us; skew 500 ppm
Jun 19 18:42:28 dist-test-slave-mld5 krb5kdc[23148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894548, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:28.687877 23265 init.cc:377] Logged in from keytab as kudu/127.21.143.65@KRBTEST.COM (short username kudu)
I20260619 18:42:28.688722 23265 webserver.cc:460] Webserver started at http://127.21.143.65:39517/ using document root <none> and password file <none>
I20260619 18:42:28.689303 23265 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:28.689397 23265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:28.689637 23265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:28.691793 23265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-0/data/instance:
uuid: "b398b7c2ea92406190c76ed6d810a151"
format_stamp: "Formatted at 2026-06-19 18:42:28 on dist-test-slave-mld5"
server_key: "a1afd8d790e5356f8fe1c7cb66913797"
server_key_iv: "7a5e1b7b06103620a718ed433328f875"
server_key_version: "encryptionkey@0"
I20260619 18:42:28.692379 23265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance:
uuid: "b398b7c2ea92406190c76ed6d810a151"
format_stamp: "Formatted at 2026-06-19 18:42:28 on dist-test-slave-mld5"
server_key: "a1afd8d790e5356f8fe1c7cb66913797"
server_key_iv: "7a5e1b7b06103620a718ed433328f875"
server_key_version: "encryptionkey@0"
I20260619 18:42:28.696260 23265 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 18:42:28.698683 23280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:28.699975 23265 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 18:42:28.700130 23265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-0/wal
uuid: "b398b7c2ea92406190c76ed6d810a151"
format_stamp: "Formatted at 2026-06-19 18:42:28 on dist-test-slave-mld5"
server_key: "a1afd8d790e5356f8fe1c7cb66913797"
server_key_iv: "7a5e1b7b06103620a718ed433328f875"
server_key_version: "encryptionkey@0"
I20260619 18:42:28.700249 23265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:28.726567 23265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:28.729979 23265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:28.730192 23265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:28.731769 23265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:28.731841 23265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:28.731913 23265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:28.731958 23265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:42:28 dist-test-slave-mld5 krb5kdc[23148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894548, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:28.742784 23265 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.65:43707
I20260619 18:42:28.743005 23395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.65:43707 every 8 connection(s)
I20260619 18:42:28.744001 23265 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
I20260619 18:42:28.744150 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 23265
I20260619 18:42:28.744269 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance
I20260619 18:42:28.744621 22077 external_mini_cluster.cc:1428] Setting key 8b85f2fdbacf1f45a5cbede14cbb1dbd
I20260619 18:42:28.746018 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:28.732506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:49677 (local address 127.21.143.126:45809)
0619 18:42:28.732682 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:28.732693 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:28.733949 (+ 1256us) server_negotiation.cc:407] Connection header received
0619 18:42:28.734979 (+ 1030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:28.734982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:28.735037 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:28.735128 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:28.736851 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:28.737384 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:28.738701 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:28.739123 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:28.739253 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:28.741899 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:28.741903 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:28.741906 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:28.741935 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:28.743612 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:28.744171 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:28.744174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:28.744175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:28.744286 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:28.744746 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:28.744750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:28.744752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:28.744983 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:28.745099 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:28.745713 (+ 614us) server_negotiation.cc:299] Negotiation successful
0619 18:42:28.745849 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 18:42:28.747004 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:28.733315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:45809 (local address 127.21.143.65:49677)
0619 18:42:28.733762 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:28.733835 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:28.734774 (+ 939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:28.735367 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:28.735378 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:28.735893 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:28.736662 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:28.736687 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:28.737518 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:28.737522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:28.738553 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:28.738572 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:28.739316 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:28.739322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:28.739517 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:28.740140 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:28.740156 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:28.741717 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:28.743723 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:28.743727 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:28.743738 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:28.744018 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:28.744427 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:28.744429 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:28.744431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:28.744618 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:28.745155 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:28.745160 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:28.745456 (+ 296us) client_negotiation.cc:769] Sending connection context
0619 18:42:28.745668 (+ 212us) client_negotiation.cc:240] Negotiation successful
0619 18:42:28.745995 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":130,"threads_started":1}
I20260619 18:42:28.748399 23396 heartbeater.cc:347] Connected to a master server at 127.21.143.126:45809
I20260619 18:42:28.748543 23396 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:28.749033 23396 heartbeater.cc:511] Master 127.21.143.126:45809 requested a full tablet report, sending...
I20260619 18:42:28.750700 23192 ts_manager.cc:194] Registered new tserver with Master: b398b7c2ea92406190c76ed6d810a151 (127.21.143.65:43707)
I20260619 18:42:28.752099 23192 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.65@KRBTEST.COM'} at 127.21.143.65:49677
WARNING: no policy specified for kudu/127.21.143.66@KRBTEST.COM; defaulting to no policy
I20260619 18:42:28.758337 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:28.750710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:51231 (local address 127.21.143.126:45809)
0619 18:42:28.750821 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:28.750826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:28.751157 (+ 331us) server_negotiation.cc:407] Connection header received
0619 18:42:28.751361 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:28.751364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:28.751409 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:28.751494 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:28.752017 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:28.752652 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:28.753607 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:28.753972 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:28.754068 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:28.755327 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:28.755331 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:28.755334 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:28.755357 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:28.756747 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:28.757180 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:28.757183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:28.757185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:28.757266 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:28.757542 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:28.757546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:28.757548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:28.757723 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:28.757829 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:28.758082 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 18:42:28.758187 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 18:42:28.758352 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:28.750639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:45809 (local address 127.21.143.65:51231)
0619 18:42:28.751061 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:28.751078 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:28.751231 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:28.751534 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:28.751538 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:28.751751 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:28.751888 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:28.751896 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:28.752792 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:28.752795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:28.753487 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:28.753496 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:28.754071 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:28.754074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:28.754179 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:28.754695 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:28.754712 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:28.755158 (+ 446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:28.756865 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:28.756869 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:28.756871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:28.757061 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:28.757364 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:28.757367 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:28.757369 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:28.757452 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:28.757820 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:28.757823 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:28.757916 (+ 93us) client_negotiation.cc:769] Sending connection context
0619 18:42:28.758019 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 18:42:28.758162 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":88,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab.
Entry for principal kudu/127.21.143.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab.
I20260619 18:42:28.772482 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.66:0
--local_ip_for_outbound_sockets=127.21.143.66
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:45809
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:37357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:28.880947 23405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:28.881212 23405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:28.881292 23405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:28.885170 23405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:28.885249 23405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:28.885319 23405 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:42:28.885396 23405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.66
I20260619 18:42:28.890298 23405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:37357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.143.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.143.126:45809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.66
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:28.891270 23405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:28.892246 23405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:28.901425 23410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:28.901439 23411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:28.901582 23405 server_base.cc:1034] running on GCE node
W20260619 18:42:28.901439 23413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:28.902064 23405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:28.902633 23405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:28.903803 23405 hybrid_clock.cc:648] HybridClock initialized: now 1781894548903783 us; error 31 us; skew 500 ppm
Jun 19 18:42:28 dist-test-slave-mld5 krb5kdc[23148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894548, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:28.906428 23405 init.cc:377] Logged in from keytab as kudu/127.21.143.66@KRBTEST.COM (short username kudu)
I20260619 18:42:28.907193 23405 webserver.cc:460] Webserver started at http://127.21.143.66:43639/ using document root <none> and password file <none>
I20260619 18:42:28.907763 23405 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:28.907820 23405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:28.907977 23405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:28.909873 23405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-1/data/instance:
uuid: "72b20147812547e08d6ee9cda7644bfa"
format_stamp: "Formatted at 2026-06-19 18:42:28 on dist-test-slave-mld5"
server_key: "3013bef5a1a7bce62f4011be74427e23"
server_key_iv: "3fe71ed6dd675f3453ba3a311ee775de"
server_key_version: "encryptionkey@0"
I20260619 18:42:28.910391 23405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance:
uuid: "72b20147812547e08d6ee9cda7644bfa"
format_stamp: "Formatted at 2026-06-19 18:42:28 on dist-test-slave-mld5"
server_key: "3013bef5a1a7bce62f4011be74427e23"
server_key_iv: "3fe71ed6dd675f3453ba3a311ee775de"
server_key_version: "encryptionkey@0"
I20260619 18:42:28.914008 23405 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 18:42:28.916417 23420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:28.917603 23405 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:42:28.917701 23405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-1/wal
uuid: "72b20147812547e08d6ee9cda7644bfa"
format_stamp: "Formatted at 2026-06-19 18:42:28 on dist-test-slave-mld5"
server_key: "3013bef5a1a7bce62f4011be74427e23"
server_key_iv: "3fe71ed6dd675f3453ba3a311ee775de"
server_key_version: "encryptionkey@0"
I20260619 18:42:28.917784 23405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:28.935900 23405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:28.938835 23405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:28.938992 23405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:28.940457 23405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:28.940507 23405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:28.940549 23405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:28.940610 23405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:42:28 dist-test-slave-mld5 krb5kdc[23148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894548, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:28.952070 23405 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.66:39133
I20260619 18:42:28.952095 23535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.66:39133 every 8 connection(s)
I20260619 18:42:28.952940 23405 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
I20260619 18:42:28.954644 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:28.941417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:56161 (local address 127.21.143.126:45809)
0619 18:42:28.941547 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:28.941550 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:28.942660 (+ 1110us) server_negotiation.cc:407] Connection header received
0619 18:42:28.943892 (+ 1232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:28.943895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:28.943961 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:28.944065 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:28.945389 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:28.945924 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:28.947342 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:28.947625 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:28.947737 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:28.950738 (+ 3001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:28.950742 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:28.950747 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:28.950777 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:28.952382 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:28.952906 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:28.952910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:28.952912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:28.953024 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:28.953404 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:28.953408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:28.953410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:28.953670 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:28.953760 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:28.954319 (+ 559us) server_negotiation.cc:299] Negotiation successful
0619 18:42:28.954467 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 18:42:28.955585 23429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:28.942018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:45809 (local address 127.21.143.66:56161)
0619 18:42:28.942503 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:28.942546 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:28.943637 (+ 1091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:28.944252 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:28.944260 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:28.944685 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:28.945245 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:28.945260 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:28.946056 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:28.946064 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:28.947203 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:28.947213 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:28.947791 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:28.947796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:28.948001 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:28.948692 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:28.948721 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:28.950566 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:28.952503 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:28.952509 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:28.952523 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:28.952791 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:28.953124 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:28.953128 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:28.953133 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:28.953300 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:28.953767 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:28.953776 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:28.954057 (+ 281us) client_negotiation.cc:769] Sending connection context
0619 18:42:28.954258 (+ 201us) client_negotiation.cc:240] Negotiation successful
0619 18:42:28.954634 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":139,"threads_started":1}
I20260619 18:42:28.956712 23536 heartbeater.cc:347] Connected to a master server at 127.21.143.126:45809
I20260619 18:42:28.956872 23536 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:28.957384 23536 heartbeater.cc:511] Master 127.21.143.126:45809 requested a full tablet report, sending...
I20260619 18:42:28.958437 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 23405
I20260619 18:42:28.958494 23193 ts_manager.cc:194] Registered new tserver with Master: 72b20147812547e08d6ee9cda7644bfa (127.21.143.66:39133)
I20260619 18:42:28.958547 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance
I20260619 18:42:28.958835 22077 external_mini_cluster.cc:1428] Setting key 1a3994df8b8d96cc056a3b945e685409
I20260619 18:42:28.959240 23193 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.66@KRBTEST.COM'} at 127.21.143.66:56161
I20260619 18:42:28.967293 23542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:28.959601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:45809 (local address 127.21.143.66:39699)
0619 18:42:28.959842 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:28.959857 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:28.959982 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:28.960265 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:28.960268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:28.960469 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:28.960579 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:28.960587 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:28.961630 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:28.961632 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:28.962586 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:28.962601 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:28.963072 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:28.963075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:28.963183 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:28.963726 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:28.963737 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:28.964038 (+ 301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:28.965769 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:28.965773 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:28.965775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:28.965968 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:28.966250 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:28.966255 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:28.966257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:28.966341 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:28.966749 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:28.966752 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:28.966846 (+ 94us) client_negotiation.cc:769] Sending connection context
0619 18:42:28.966950 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 18:42:28.967092 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":97,"threads_started":1}
I20260619 18:42:28.967284 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:28.959568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:39699 (local address 127.21.143.126:45809)
0619 18:42:28.959697 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:28.959704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:28.959920 (+ 216us) server_negotiation.cc:407] Connection header received
0619 18:42:28.960096 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:28.960100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:28.960147 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:28.960227 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:28.960708 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:28.961505 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:28.962747 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:28.962950 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:28.963058 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:28.964187 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:28.964191 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:28.964193 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:28.964215 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:28.965658 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:28.966072 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:28.966077 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:28.966078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:28.966161 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:28.966430 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:28.966434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:28.966436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:28.966649 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:28.966760 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:28.967023 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 18:42:28.967131 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
WARNING: no policy specified for kudu/127.21.143.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab.
Entry for principal kudu/127.21.143.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab.
I20260619 18:42:28.984608 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.67:0
--local_ip_for_outbound_sockets=127.21.143.67
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:45809
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:37357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:29.092510 23545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:29.092758 23545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:29.092854 23545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:29.096988 23545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:29.097059 23545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:29.097126 23545 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:42:29.097204 23545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.67
I20260619 18:42:29.102391 23545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:37357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.143.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.143.126:45809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.67
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:29.103386 23545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:29.104301 23545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:29.113328 23551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:29.113335 23550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:29.113335 23553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:29.113371 23545 server_base.cc:1034] running on GCE node
I20260619 18:42:29.113986 23545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:29.114607 23545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:29.115777 23545 hybrid_clock.cc:648] HybridClock initialized: now 1781894549115744 us; error 43 us; skew 500 ppm
Jun 19 18:42:29 dist-test-slave-mld5 krb5kdc[23148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894549, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:29.118449 23545 init.cc:377] Logged in from keytab as kudu/127.21.143.67@KRBTEST.COM (short username kudu)
I20260619 18:42:29.119249 23545 webserver.cc:460] Webserver started at http://127.21.143.67:42715/ using document root <none> and password file <none>
I20260619 18:42:29.119858 23545 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:29.119917 23545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:29.120085 23545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:29.122051 23545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-2/data/instance:
uuid: "67b59204f8214e069ec7c042cae0733c"
format_stamp: "Formatted at 2026-06-19 18:42:29 on dist-test-slave-mld5"
server_key: "e46f6284748cdd0c34e6eefc64cdf9ed"
server_key_iv: "e22d2ce4f58e5fd8332794cac8a4fe84"
server_key_version: "encryptionkey@0"
I20260619 18:42:29.122619 23545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance:
uuid: "67b59204f8214e069ec7c042cae0733c"
format_stamp: "Formatted at 2026-06-19 18:42:29 on dist-test-slave-mld5"
server_key: "e46f6284748cdd0c34e6eefc64cdf9ed"
server_key_iv: "e22d2ce4f58e5fd8332794cac8a4fe84"
server_key_version: "encryptionkey@0"
I20260619 18:42:29.126077 23545 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 18:42:29.128312 23560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:29.129437 23545 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:42:29.129577 23545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-2/wal
uuid: "67b59204f8214e069ec7c042cae0733c"
format_stamp: "Formatted at 2026-06-19 18:42:29 on dist-test-slave-mld5"
server_key: "e46f6284748cdd0c34e6eefc64cdf9ed"
server_key_iv: "e22d2ce4f58e5fd8332794cac8a4fe84"
server_key_version: "encryptionkey@0"
I20260619 18:42:29.129678 23545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:29.156256 23545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:29.159684 23545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:29.159889 23545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:29.161351 23545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:29.161423 23545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:29.161496 23545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:29.161541 23545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:42:29 dist-test-slave-mld5 krb5kdc[23148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894549, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:29.172245 23545 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.67:43603
I20260619 18:42:29.172339 23675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.67:43603 every 8 connection(s)
I20260619 18:42:29.173385 23545 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
I20260619 18:42:29.175877 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:29.162273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:60141 (local address 127.21.143.126:45809)
0619 18:42:29.162397 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:29.162401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:29.163555 (+ 1154us) server_negotiation.cc:407] Connection header received
0619 18:42:29.164654 (+ 1099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:29.164657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:29.164719 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:29.164812 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:29.166422 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.167272 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.169094 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.169429 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.169550 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:29.172136 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:29.172142 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:29.172149 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:29.172185 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:29.173917 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:29.174501 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:29.174504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:29.174505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:29.174581 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:29.174864 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:29.174866 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:29.174867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:29.175047 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:29.175125 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:29.175664 (+ 539us) server_negotiation.cc:299] Negotiation successful
0619 18:42:29.175763 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 18:42:29.176664 23569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:29.162863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:45809 (local address 127.21.143.67:60141)
0619 18:42:29.163406 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:29.163450 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:29.164404 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:29.165046 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:29.165056 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:29.165539 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:29.166242 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:29.166266 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.167452 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.167457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:29.168961 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:29.168972 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.169587 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.169592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:29.169779 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:29.170410 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:29.170431 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:29.171970 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:29.174050 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:29.174055 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:29.174071 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:29.174387 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:29.174666 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:29.174669 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:29.174670 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:29.174790 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:29.175155 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:29.175160 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:29.175452 (+ 292us) client_negotiation.cc:769] Sending connection context
0619 18:42:29.175615 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 18:42:29.175844 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":143,"threads_started":1}
I20260619 18:42:29.177829 23676 heartbeater.cc:347] Connected to a master server at 127.21.143.126:45809
I20260619 18:42:29.177980 23676 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:29.178434 23676 heartbeater.cc:511] Master 127.21.143.126:45809 requested a full tablet report, sending...
I20260619 18:42:29.179430 23192 ts_manager.cc:194] Registered new tserver with Master: 67b59204f8214e069ec7c042cae0733c (127.21.143.67:43603)
I20260619 18:42:29.180131 23192 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.67@KRBTEST.COM'} at 127.21.143.67:60141
I20260619 18:42:29.180629 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 23545
I20260619 18:42:29.180757 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance
I20260619 18:42:29.181063 22077 external_mini_cluster.cc:1428] Setting key ce4548ae5ea6f7261eccc4d64ee7d3c7
I20260619 18:42:29.182940 22077 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:42:29.187212 23682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:29.179828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:45809 (local address 127.21.143.67:45401)
0619 18:42:29.180107 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:29.180121 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:29.180224 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:29.180562 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:29.180564 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:29.180753 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:29.180859 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:29.180865 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.181825 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.181846 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:29.182551 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:29.182560 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.183014 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.183018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:29.183105 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:29.183638 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:29.183651 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:29.183923 (+ 272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:29.185799 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:29.185801 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:29.185803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:29.185980 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:29.186298 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:29.186301 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:29.186302 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:29.186383 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:29.186830 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:29.186832 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:29.186900 (+ 68us) client_negotiation.cc:769] Sending connection context
0619 18:42:29.186980 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 18:42:29.187085 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":99,"threads_started":1}
I20260619 18:42:29.187458 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:29.179941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:45401 (local address 127.21.143.126:45809)
0619 18:42:29.180056 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:29.180060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:29.180196 (+ 136us) server_negotiation.cc:407] Connection header received
0619 18:42:29.180318 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:29.180321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:29.180386 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:29.180457 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:29.180978 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.181624 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.182668 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.182908 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.183012 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:29.184050 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:29.184054 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:29.184056 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:29.184079 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:29.185682 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:29.186074 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:29.186080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:29.186085 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:29.186175 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:29.186482 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:29.186488 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:29.186493 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:29.186736 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:29.186838 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:29.187062 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 18:42:29.187190 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 18:42:29 dist-test-slave-mld5 krb5kdc[23148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894549, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 18:42:29 dist-test-slave-mld5 krb5kdc[23148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894549, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:29.208942 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:29.200686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48810 (local address 127.21.143.126:45809)
0619 18:42:29.200804 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:29.200808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:29.200818 (+ 10us) server_negotiation.cc:407] Connection header received
0619 18:42:29.200853 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:29.200855 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:29.200888 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:29.200954 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:29.201584 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.202080 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.202860 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.203070 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.203147 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:29.205822 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:29.205825 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:29.205828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:29.205848 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:29.206980 (+ 1132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:29.207597 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:29.207601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:29.207603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:29.207683 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:29.207970 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:29.207973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:29.207975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:29.208144 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:29.208298 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:29.208544 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 18:42:29.208777 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:42:29.211830 23192 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:42:29.214818 23192 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:42:29.228832 23693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:29.223113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:43603 (local address 127.0.0.1:48364)
0619 18:42:29.224175 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:29.224188 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:29.224262 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:29.224914 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:29.224920 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:29.224943 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:29.225184 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:29.225191 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.226212 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.226215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:29.227544 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:29.227553 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.228432 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.228435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:29.228520 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:29.228549 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 18:42:29.228624 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 18:42:29.228717 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1000,"mutex_wait_us":109,"thread_start_us":105,"threads_started":1}
I20260619 18:42:29.228893 23697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:29.223830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48364 (local address 127.21.143.67:43603)
0619 18:42:29.224257 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:29.224275 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:29.224299 (+ 24us) server_negotiation.cc:407] Connection header received
0619 18:42:29.224378 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:29.224383 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:29.224609 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:29.224750 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:29.225318 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.226079 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.227667 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.228322 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.228431 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:29.228596 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:29.228653 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 18:42:29.228748 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":106,"threads_started":1}
I20260619 18:42:29.229399 23696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:29.223776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:43707 (local address 127.0.0.1:51636)
0619 18:42:29.224518 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:29.224533 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:29.224621 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:29.225155 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:29.225159 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:29.225176 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:29.225260 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:29.225266 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.226684 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.226687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:29.227982 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:29.227993 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.229023 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.229028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:29.229117 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:29.229138 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 18:42:29.229214 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 18:42:29.229282 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"mutex_wait_us":75,"thread_start_us":84,"threads_started":1}
I20260619 18:42:29.229476 23698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:29.223937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51636 (local address 127.21.143.65:43707)
0619 18:42:29.224368 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:29.224384 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:29.224720 (+ 336us) server_negotiation.cc:407] Connection header received
0619 18:42:29.224801 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:29.224813 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:29.225003 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:29.225165 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:29.225371 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.226485 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.228122 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.228868 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.228976 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:29.229139 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:29.229256 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 18:42:29.229348 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":60,"threads_started":1}
I20260619 18:42:29.230692 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:29.223035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:39133 (local address 127.0.0.1:51210)
0619 18:42:29.223787 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:29.223835 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:29.223966 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:29.224935 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:29.224938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:29.224957 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:29.225158 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:29.225166 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.226604 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.226608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:29.228001 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:29.228007 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.230315 (+ 2308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.230319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:29.230413 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:29.230434 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 18:42:29.230504 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 18:42:29.230569 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":154,"spinlock_wait_cycles":768,"thread_start_us":96,"threads_started":1}
I20260619 18:42:29.231073 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:29.223151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51210 (local address 127.21.143.66:39133)
0619 18:42:29.224508 (+ 1357us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:29.224519 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:29.224539 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:42:29.224604 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:29.224609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:29.224794 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:29.224936 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:29.225290 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.226485 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.229216 (+ 2731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.229993 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.230106 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:29.230250 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:29.230855 (+ 605us) server_negotiation.cc:299] Negotiation successful
0619 18:42:29.230945 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1244,"thread_start_us":99,"threads_started":1}
I20260619 18:42:29.232275 23610 tablet_service.cc:1467] Processing CreateTablet for tablet c64a424847d04029a50baf00f6cb1eed (DEFAULT_TABLE table=test-table [id=dbdc82c91fa749758da86101f2076cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:42:29.232280 23330 tablet_service.cc:1467] Processing CreateTablet for tablet c64a424847d04029a50baf00f6cb1eed (DEFAULT_TABLE table=test-table [id=dbdc82c91fa749758da86101f2076cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:42:29.233279 23330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c64a424847d04029a50baf00f6cb1eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:29.233299 23610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c64a424847d04029a50baf00f6cb1eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:29.233438 23470 tablet_service.cc:1467] Processing CreateTablet for tablet c64a424847d04029a50baf00f6cb1eed (DEFAULT_TABLE table=test-table [id=dbdc82c91fa749758da86101f2076cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:42:29.234270 23470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c64a424847d04029a50baf00f6cb1eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:29.239725 23699 tablet_bootstrap.cc:492] T c64a424847d04029a50baf00f6cb1eed P 67b59204f8214e069ec7c042cae0733c: Bootstrap starting.
I20260619 18:42:29.241266 23701 tablet_bootstrap.cc:492] T c64a424847d04029a50baf00f6cb1eed P 72b20147812547e08d6ee9cda7644bfa: Bootstrap starting.
I20260619 18:42:29.241647 23700 tablet_bootstrap.cc:492] T c64a424847d04029a50baf00f6cb1eed P b398b7c2ea92406190c76ed6d810a151: Bootstrap starting.
I20260619 18:42:29.241990 23699 tablet_bootstrap.cc:654] T c64a424847d04029a50baf00f6cb1eed P 67b59204f8214e069ec7c042cae0733c: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:29.242873 23699 log.cc:824] T c64a424847d04029a50baf00f6cb1eed P 67b59204f8214e069ec7c042cae0733c: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:29.243448 23701 tablet_bootstrap.cc:654] T c64a424847d04029a50baf00f6cb1eed P 72b20147812547e08d6ee9cda7644bfa: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:29.243891 23700 tablet_bootstrap.cc:654] T c64a424847d04029a50baf00f6cb1eed P b398b7c2ea92406190c76ed6d810a151: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:29.244244 23701 log.cc:824] T c64a424847d04029a50baf00f6cb1eed P 72b20147812547e08d6ee9cda7644bfa: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:29.244721 23700 log.cc:824] T c64a424847d04029a50baf00f6cb1eed P b398b7c2ea92406190c76ed6d810a151: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:29.245000 23699 tablet_bootstrap.cc:492] T c64a424847d04029a50baf00f6cb1eed P 67b59204f8214e069ec7c042cae0733c: No bootstrap required, opened a new log
I20260619 18:42:29.245215 23699 ts_tablet_manager.cc:1397] T c64a424847d04029a50baf00f6cb1eed P 67b59204f8214e069ec7c042cae0733c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 18:42:29.246143 23701 tablet_bootstrap.cc:492] T c64a424847d04029a50baf00f6cb1eed P 72b20147812547e08d6ee9cda7644bfa: No bootstrap required, opened a new log
I20260619 18:42:29.246316 23701 ts_tablet_manager.cc:1397] T c64a424847d04029a50baf00f6cb1eed P 72b20147812547e08d6ee9cda7644bfa: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260619 18:42:29.246615 23700 tablet_bootstrap.cc:492] T c64a424847d04029a50baf00f6cb1eed P b398b7c2ea92406190c76ed6d810a151: No bootstrap required, opened a new log
I20260619 18:42:29.246780 23700 ts_tablet_manager.cc:1397] T c64a424847d04029a50baf00f6cb1eed P b398b7c2ea92406190c76ed6d810a151: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 18:42:29.248428 23699 raft_consensus.cc:348] T c64a424847d04029a50baf00f6cb1eed P 67b59204f8214e069ec7c042cae0733c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } } peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } }
I20260619 18:42:29.248723 23699 raft_consensus.cc:374] T c64a424847d04029a50baf00f6cb1eed P 67b59204f8214e069ec7c042cae0733c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:29.248826 23699 raft_consensus.cc:729] T c64a424847d04029a50baf00f6cb1eed P 67b59204f8214e069ec7c042cae0733c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b59204f8214e069ec7c042cae0733c, State: Initialized, Role: FOLLOWER
I20260619 18:42:29.249264 23699 consensus_queue.cc:260] T c64a424847d04029a50baf00f6cb1eed P 67b59204f8214e069ec7c042cae0733c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } } peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } }
I20260619 18:42:29.249926 23676 heartbeater.cc:503] Master 127.21.143.126:45809 was elected leader, sending a full tablet report...
I20260619 18:42:29.250233 23699 ts_tablet_manager.cc:1428] T c64a424847d04029a50baf00f6cb1eed P 67b59204f8214e069ec7c042cae0733c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 18:42:29.250221 23701 raft_consensus.cc:348] T c64a424847d04029a50baf00f6cb1eed P 72b20147812547e08d6ee9cda7644bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } } peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } }
I20260619 18:42:29.250451 23701 raft_consensus.cc:374] T c64a424847d04029a50baf00f6cb1eed P 72b20147812547e08d6ee9cda7644bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:29.250502 23701 raft_consensus.cc:729] T c64a424847d04029a50baf00f6cb1eed P 72b20147812547e08d6ee9cda7644bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72b20147812547e08d6ee9cda7644bfa, State: Initialized, Role: FOLLOWER
I20260619 18:42:29.250406 23700 raft_consensus.cc:348] T c64a424847d04029a50baf00f6cb1eed P b398b7c2ea92406190c76ed6d810a151 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } } peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } }
I20260619 18:42:29.250631 23700 raft_consensus.cc:374] T c64a424847d04029a50baf00f6cb1eed P b398b7c2ea92406190c76ed6d810a151 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:29.250687 23700 raft_consensus.cc:729] T c64a424847d04029a50baf00f6cb1eed P b398b7c2ea92406190c76ed6d810a151 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b398b7c2ea92406190c76ed6d810a151, State: Initialized, Role: FOLLOWER
I20260619 18:42:29.251063 23700 consensus_queue.cc:260] T c64a424847d04029a50baf00f6cb1eed P b398b7c2ea92406190c76ed6d810a151 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } } peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } }
I20260619 18:42:29.251137 23701 consensus_queue.cc:260] T c64a424847d04029a50baf00f6cb1eed P 72b20147812547e08d6ee9cda7644bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } } peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } }
I20260619 18:42:29.252027 23700 ts_tablet_manager.cc:1428] T c64a424847d04029a50baf00f6cb1eed P b398b7c2ea92406190c76ed6d810a151: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 18:42:29.252027 23701 ts_tablet_manager.cc:1428] T c64a424847d04029a50baf00f6cb1eed P 72b20147812547e08d6ee9cda7644bfa: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 18:42:29.252113 23536 heartbeater.cc:503] Master 127.21.143.126:45809 was elected leader, sending a full tablet report...
I20260619 18:42:29.253238 23396 heartbeater.cc:503] Master 127.21.143.126:45809 was elected leader, sending a full tablet report...
I20260619 18:42:29.296923 23707 raft_consensus.cc:482] T c64a424847d04029a50baf00f6cb1eed P 72b20147812547e08d6ee9cda7644bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:42:29.297124 23707 raft_consensus.cc:504] T c64a424847d04029a50baf00f6cb1eed P 72b20147812547e08d6ee9cda7644bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } } peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } }
I20260619 18:42:29.298425 23707 leader_election.cc:288] T c64a424847d04029a50baf00f6cb1eed P 72b20147812547e08d6ee9cda7644bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b59204f8214e069ec7c042cae0733c (127.21.143.67:43603), b398b7c2ea92406190c76ed6d810a151 (127.21.143.65:43707)
I20260619 18:42:29.302517 23429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:29.298663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:43603 (local address 127.21.143.66:53507)
0619 18:42:29.298826 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:29.298845 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:29.298937 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:29.299192 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:29.299195 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:29.299212 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:29.299386 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:29.299393 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.300519 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.300522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:29.301516 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:29.301526 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.302188 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.302190 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:29.302280 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:29.302300 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 18:42:29.302359 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 18:42:29.302413 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 18:42:29.302601 23697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:29.298785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:53507 (local address 127.21.143.67:43603)
0619 18:42:29.298925 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:29.298930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:29.298948 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:42:29.299023 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:29.299026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:29.299067 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:29.299167 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:29.299538 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.300378 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.301648 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.302069 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.302189 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:29.302242 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:29.302397 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 18:42:29.302469 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 18:42:29.303241 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:29.298667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:43707 (local address 127.21.143.66:38377)
0619 18:42:29.299382 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:29.299398 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:29.299497 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:29.299774 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:29.299777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:29.299794 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:29.300370 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:29.300378 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.301071 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.301074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:29.301982 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:29.301989 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.302840 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.302844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:29.302948 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:29.302966 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 18:42:29.303026 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 18:42:29.303092 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":340,"thread_start_us":127,"threads_started":1}
I20260619 18:42:29.303253 23698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:29.298764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:38377 (local address 127.21.143.65:43707)
0619 18:42:29.298933 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:29.298938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:29.299474 (+ 536us) server_negotiation.cc:407] Connection header received
0619 18:42:29.299629 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:29.299638 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:29.299679 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:29.299738 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:29.300484 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.300963 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.302105 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.302649 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.302773 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:29.302819 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:29.303069 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 18:42:29.303143 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260619 18:42:29.303236 23630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c64a424847d04029a50baf00f6cb1eed" candidate_uuid: "72b20147812547e08d6ee9cda7644bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b59204f8214e069ec7c042cae0733c" is_pre_election: true
I20260619 18:42:29.303574 23630 raft_consensus.cc:2454] T c64a424847d04029a50baf00f6cb1eed P 67b59204f8214e069ec7c042cae0733c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72b20147812547e08d6ee9cda7644bfa in term 0.
I20260619 18:42:29.303902 23350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c64a424847d04029a50baf00f6cb1eed" candidate_uuid: "72b20147812547e08d6ee9cda7644bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b398b7c2ea92406190c76ed6d810a151" is_pre_election: true
I20260619 18:42:29.304116 23423 leader_election.cc:302] T c64a424847d04029a50baf00f6cb1eed P 72b20147812547e08d6ee9cda7644bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b59204f8214e069ec7c042cae0733c, 72b20147812547e08d6ee9cda7644bfa; no voters:
I20260619 18:42:29.304198 23350 raft_consensus.cc:2454] T c64a424847d04029a50baf00f6cb1eed P b398b7c2ea92406190c76ed6d810a151 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72b20147812547e08d6ee9cda7644bfa in term 0.
I20260619 18:42:29.304426 23707 raft_consensus.cc:2781] T c64a424847d04029a50baf00f6cb1eed P 72b20147812547e08d6ee9cda7644bfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:42:29.304548 23707 raft_consensus.cc:482] T c64a424847d04029a50baf00f6cb1eed P 72b20147812547e08d6ee9cda7644bfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:42:29.304606 23707 raft_consensus.cc:3037] T c64a424847d04029a50baf00f6cb1eed P 72b20147812547e08d6ee9cda7644bfa [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:29.305717 23707 raft_consensus.cc:504] T c64a424847d04029a50baf00f6cb1eed P 72b20147812547e08d6ee9cda7644bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } } peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } }
I20260619 18:42:29.306053 23707 leader_election.cc:288] T c64a424847d04029a50baf00f6cb1eed P 72b20147812547e08d6ee9cda7644bfa [CANDIDATE]: Term 1 election: Requested vote from peers 67b59204f8214e069ec7c042cae0733c (127.21.143.67:43603), b398b7c2ea92406190c76ed6d810a151 (127.21.143.65:43707)
I20260619 18:42:29.306352 23630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c64a424847d04029a50baf00f6cb1eed" candidate_uuid: "72b20147812547e08d6ee9cda7644bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b59204f8214e069ec7c042cae0733c"
I20260619 18:42:29.306417 23350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c64a424847d04029a50baf00f6cb1eed" candidate_uuid: "72b20147812547e08d6ee9cda7644bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b398b7c2ea92406190c76ed6d810a151"
I20260619 18:42:29.306475 23630 raft_consensus.cc:3037] T c64a424847d04029a50baf00f6cb1eed P 67b59204f8214e069ec7c042cae0733c [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:29.306511 23350 raft_consensus.cc:3037] T c64a424847d04029a50baf00f6cb1eed P b398b7c2ea92406190c76ed6d810a151 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:29.307806 23350 raft_consensus.cc:2454] T c64a424847d04029a50baf00f6cb1eed P b398b7c2ea92406190c76ed6d810a151 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72b20147812547e08d6ee9cda7644bfa in term 1.
I20260619 18:42:29.307806 23630 raft_consensus.cc:2454] T c64a424847d04029a50baf00f6cb1eed P 67b59204f8214e069ec7c042cae0733c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72b20147812547e08d6ee9cda7644bfa in term 1.
I20260619 18:42:29.308148 23422 leader_election.cc:302] T c64a424847d04029a50baf00f6cb1eed P 72b20147812547e08d6ee9cda7644bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72b20147812547e08d6ee9cda7644bfa, b398b7c2ea92406190c76ed6d810a151; no voters:
I20260619 18:42:29.308305 23707 raft_consensus.cc:2781] T c64a424847d04029a50baf00f6cb1eed P 72b20147812547e08d6ee9cda7644bfa [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:29.308503 23707 raft_consensus.cc:686] T c64a424847d04029a50baf00f6cb1eed P 72b20147812547e08d6ee9cda7644bfa [term 1 LEADER]: Becoming Leader. State: Replica: 72b20147812547e08d6ee9cda7644bfa, State: Running, Role: LEADER
I20260619 18:42:29.308854 23707 consensus_queue.cc:237] T c64a424847d04029a50baf00f6cb1eed P 72b20147812547e08d6ee9cda7644bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } } peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } }
I20260619 18:42:29.311947 23191 catalog_manager.cc:5495] T c64a424847d04029a50baf00f6cb1eed P 72b20147812547e08d6ee9cda7644bfa reported cstate change: term changed from 0 to 1, leader changed from <none> to 72b20147812547e08d6ee9cda7644bfa (127.21.143.66). New cstate: current_term: 1 leader_uuid: "72b20147812547e08d6ee9cda7644bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } health_report { overall_health: UNKNOWN } } }
I20260619 18:42:29.330099 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:29.326024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51212 (local address 127.21.143.66:39133)
0619 18:42:29.326158 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:29.326173 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:29.326188 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:42:29.326408 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:29.326411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:29.326470 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:29.326566 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:42:29.326837 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.327414 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.328345 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.328575 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.328699 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:29.329089 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:42:29.329629 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:42:29.329740 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:29.329896 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 18:42:29.329960 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 18:42:29.336385 23350 raft_consensus.cc:1261] T c64a424847d04029a50baf00f6cb1eed P b398b7c2ea92406190c76ed6d810a151 [term 1 FOLLOWER]: Refusing update from remote peer 72b20147812547e08d6ee9cda7644bfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:42:29.336551 23630 raft_consensus.cc:1261] T c64a424847d04029a50baf00f6cb1eed P 67b59204f8214e069ec7c042cae0733c [term 1 FOLLOWER]: Refusing update from remote peer 72b20147812547e08d6ee9cda7644bfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:42:29.337142 23707 consensus_queue.cc:1035] T c64a424847d04029a50baf00f6cb1eed P 72b20147812547e08d6ee9cda7644bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:42:29.337358 23709 consensus_queue.cc:1035] T c64a424847d04029a50baf00f6cb1eed P 72b20147812547e08d6ee9cda7644bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:42:29.344931 23713 mvcc.cc:205] Tried to move back new op lower bound from 7298640074073784320 to 7298640073971744768. Current Snapshot: MvccSnapshot[applied={T|T < 7298640074073784320}]
I20260619 18:42:29.346252 23714 mvcc.cc:205] Tried to move back new op lower bound from 7298640074073784320 to 7298640073971744768. Current Snapshot: MvccSnapshot[applied={T|T < 7298640074073784320}]
I20260619 18:42:29.356390 23191 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:42:29.356588 23191 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:42:29.359160 23191 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed2f4a5d94cd421eb4f87baf37b71afa: Sending DeleteTablet for 3 replicas of tablet c64a424847d04029a50baf00f6cb1eed
I20260619 18:42:29.360005 23610 tablet_service.cc:1514] Processing DeleteTablet for tablet c64a424847d04029a50baf00f6cb1eed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:42:29 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:48364
I20260619 18:42:29.360013 23470 tablet_service.cc:1514] Processing DeleteTablet for tablet c64a424847d04029a50baf00f6cb1eed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:42:29 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:51210
I20260619 18:42:29.360198 23330 tablet_service.cc:1514] Processing DeleteTablet for tablet c64a424847d04029a50baf00f6cb1eed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:42:29 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:51636
I20260619 18:42:29.360646 23724 tablet_replica.cc:330] T c64a424847d04029a50baf00f6cb1eed P b398b7c2ea92406190c76ed6d810a151: stopping tablet replica
I20260619 18:42:29.360924 23722 tablet_replica.cc:330] T c64a424847d04029a50baf00f6cb1eed P 67b59204f8214e069ec7c042cae0733c: stopping tablet replica
I20260619 18:42:29.360986 23724 raft_consensus.cc:2229] T c64a424847d04029a50baf00f6cb1eed P b398b7c2ea92406190c76ed6d810a151 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:42:29.361222 23723 tablet_replica.cc:330] T c64a424847d04029a50baf00f6cb1eed P 72b20147812547e08d6ee9cda7644bfa: stopping tablet replica
I20260619 18:42:29.361374 23724 raft_consensus.cc:2258] T c64a424847d04029a50baf00f6cb1eed P b398b7c2ea92406190c76ed6d810a151 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:42:29.361588 23191 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 18:42:29.361706 23723 raft_consensus.cc:2229] T c64a424847d04029a50baf00f6cb1eed P 72b20147812547e08d6ee9cda7644bfa [term 1 LEADER]: Raft consensus shutting down.
W20260619 18:42:29.362015 23191 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:42:29.362041 23723 raft_consensus.cc:2258] T c64a424847d04029a50baf00f6cb1eed P 72b20147812547e08d6ee9cda7644bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:42:29.362407 23722 raft_consensus.cc:2229] T c64a424847d04029a50baf00f6cb1eed P 67b59204f8214e069ec7c042cae0733c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:42:29.362723 23722 raft_consensus.cc:2258] T c64a424847d04029a50baf00f6cb1eed P 67b59204f8214e069ec7c042cae0733c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:42:29.364454 23722 ts_tablet_manager.cc:1905] T c64a424847d04029a50baf00f6cb1eed P 67b59204f8214e069ec7c042cae0733c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:42:29.364681 23723 ts_tablet_manager.cc:1905] T c64a424847d04029a50baf00f6cb1eed P 72b20147812547e08d6ee9cda7644bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:42:29.366814 23330 tablet_service.cc:1467] Processing CreateTablet for tablet ec02a14916dc4516aebb49e65c388d59 (DEFAULT_TABLE table=test-table [id=072f84c9054445e3b3bd7e87a23ab6b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:42:29.367084 23330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec02a14916dc4516aebb49e65c388d59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:29.367074 23470 tablet_service.cc:1467] Processing CreateTablet for tablet ec02a14916dc4516aebb49e65c388d59 (DEFAULT_TABLE table=test-table [id=072f84c9054445e3b3bd7e87a23ab6b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:42:29.367174 23610 tablet_service.cc:1467] Processing CreateTablet for tablet ec02a14916dc4516aebb49e65c388d59 (DEFAULT_TABLE table=test-table [id=072f84c9054445e3b3bd7e87a23ab6b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:42:29.367319 23470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec02a14916dc4516aebb49e65c388d59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:29.367457 23610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec02a14916dc4516aebb49e65c388d59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:29.369215 23699 tablet_bootstrap.cc:492] T ec02a14916dc4516aebb49e65c388d59 P 67b59204f8214e069ec7c042cae0733c: Bootstrap starting.
I20260619 18:42:29.369455 23700 tablet_bootstrap.cc:492] T ec02a14916dc4516aebb49e65c388d59 P b398b7c2ea92406190c76ed6d810a151: Bootstrap starting.
I20260619 18:42:29.369455 23701 tablet_bootstrap.cc:492] T ec02a14916dc4516aebb49e65c388d59 P 72b20147812547e08d6ee9cda7644bfa: Bootstrap starting.
I20260619 18:42:29.370220 23699 tablet_bootstrap.cc:654] T ec02a14916dc4516aebb49e65c388d59 P 67b59204f8214e069ec7c042cae0733c: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:29.370394 23701 tablet_bootstrap.cc:654] T ec02a14916dc4516aebb49e65c388d59 P 72b20147812547e08d6ee9cda7644bfa: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:29.370394 23700 tablet_bootstrap.cc:654] T ec02a14916dc4516aebb49e65c388d59 P b398b7c2ea92406190c76ed6d810a151: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:29.371987 23723 ts_tablet_manager.cc:1918] T c64a424847d04029a50baf00f6cb1eed P 72b20147812547e08d6ee9cda7644bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:42:29.372083 23723 log.cc:1196] T c64a424847d04029a50baf00f6cb1eed P 72b20147812547e08d6ee9cda7644bfa: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-1/wal/wals/c64a424847d04029a50baf00f6cb1eed
I20260619 18:42:29.372381 23723 ts_tablet_manager.cc:1939] T c64a424847d04029a50baf00f6cb1eed P 72b20147812547e08d6ee9cda7644bfa: Deleting consensus metadata
I20260619 18:42:29.372435 23724 ts_tablet_manager.cc:1905] T c64a424847d04029a50baf00f6cb1eed P b398b7c2ea92406190c76ed6d810a151: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:42:29.373610 23178 catalog_manager.cc:4841] TS 72b20147812547e08d6ee9cda7644bfa (127.21.143.66:39133): tablet c64a424847d04029a50baf00f6cb1eed (table test-table [id=dbdc82c91fa749758da86101f2076cd5]) successfully deleted
I20260619 18:42:29.374400 23701 tablet_bootstrap.cc:492] T ec02a14916dc4516aebb49e65c388d59 P 72b20147812547e08d6ee9cda7644bfa: No bootstrap required, opened a new log
I20260619 18:42:29.374485 23701 ts_tablet_manager.cc:1397] T ec02a14916dc4516aebb49e65c388d59 P 72b20147812547e08d6ee9cda7644bfa: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260619 18:42:29.374935 23724 ts_tablet_manager.cc:1918] T c64a424847d04029a50baf00f6cb1eed P b398b7c2ea92406190c76ed6d810a151: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:42:29.375036 23724 log.cc:1196] T c64a424847d04029a50baf00f6cb1eed P b398b7c2ea92406190c76ed6d810a151: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-0/wal/wals/c64a424847d04029a50baf00f6cb1eed
I20260619 18:42:29.375001 23701 raft_consensus.cc:348] T ec02a14916dc4516aebb49e65c388d59 P 72b20147812547e08d6ee9cda7644bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } } peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } }
I20260619 18:42:29.375116 23701 raft_consensus.cc:374] T ec02a14916dc4516aebb49e65c388d59 P 72b20147812547e08d6ee9cda7644bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:29.375164 23701 raft_consensus.cc:729] T ec02a14916dc4516aebb49e65c388d59 P 72b20147812547e08d6ee9cda7644bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72b20147812547e08d6ee9cda7644bfa, State: Initialized, Role: FOLLOWER
I20260619 18:42:29.375336 23701 consensus_queue.cc:260] T ec02a14916dc4516aebb49e65c388d59 P 72b20147812547e08d6ee9cda7644bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } } peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } }
I20260619 18:42:29.375698 23701 ts_tablet_manager.cc:1428] T ec02a14916dc4516aebb49e65c388d59 P 72b20147812547e08d6ee9cda7644bfa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:42:29.375816 23724 ts_tablet_manager.cc:1939] T c64a424847d04029a50baf00f6cb1eed P b398b7c2ea92406190c76ed6d810a151: Deleting consensus metadata
I20260619 18:42:29.376091 23699 tablet_bootstrap.cc:492] T ec02a14916dc4516aebb49e65c388d59 P 67b59204f8214e069ec7c042cae0733c: No bootstrap required, opened a new log
I20260619 18:42:29.376178 23699 ts_tablet_manager.cc:1397] T ec02a14916dc4516aebb49e65c388d59 P 67b59204f8214e069ec7c042cae0733c: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260619 18:42:29.376592 23699 raft_consensus.cc:348] T ec02a14916dc4516aebb49e65c388d59 P 67b59204f8214e069ec7c042cae0733c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } } peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } }
I20260619 18:42:29.376691 23699 raft_consensus.cc:374] T ec02a14916dc4516aebb49e65c388d59 P 67b59204f8214e069ec7c042cae0733c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:29.376720 23699 raft_consensus.cc:729] T ec02a14916dc4516aebb49e65c388d59 P 67b59204f8214e069ec7c042cae0733c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b59204f8214e069ec7c042cae0733c, State: Initialized, Role: FOLLOWER
I20260619 18:42:29.376834 23699 consensus_queue.cc:260] T ec02a14916dc4516aebb49e65c388d59 P 67b59204f8214e069ec7c042cae0733c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } } peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } }
I20260619 18:42:29.377071 23699 ts_tablet_manager.cc:1428] T ec02a14916dc4516aebb49e65c388d59 P 67b59204f8214e069ec7c042cae0733c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:42:29.377063 23178 catalog_manager.cc:4841] TS b398b7c2ea92406190c76ed6d810a151 (127.21.143.65:43707): tablet c64a424847d04029a50baf00f6cb1eed (table test-table [id=dbdc82c91fa749758da86101f2076cd5]) successfully deleted
I20260619 18:42:29.377676 23722 ts_tablet_manager.cc:1918] T c64a424847d04029a50baf00f6cb1eed P 67b59204f8214e069ec7c042cae0733c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:42:29.377768 23722 log.cc:1196] T c64a424847d04029a50baf00f6cb1eed P 67b59204f8214e069ec7c042cae0733c: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-2/wal/wals/c64a424847d04029a50baf00f6cb1eed
I20260619 18:42:29.377907 23700 tablet_bootstrap.cc:492] T ec02a14916dc4516aebb49e65c388d59 P b398b7c2ea92406190c76ed6d810a151: No bootstrap required, opened a new log
I20260619 18:42:29.377986 23700 ts_tablet_manager.cc:1397] T ec02a14916dc4516aebb49e65c388d59 P b398b7c2ea92406190c76ed6d810a151: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260619 18:42:29.378075 23722 ts_tablet_manager.cc:1939] T c64a424847d04029a50baf00f6cb1eed P 67b59204f8214e069ec7c042cae0733c: Deleting consensus metadata
I20260619 18:42:29.378454 23700 raft_consensus.cc:348] T ec02a14916dc4516aebb49e65c388d59 P b398b7c2ea92406190c76ed6d810a151 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } } peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } }
I20260619 18:42:29.378557 23700 raft_consensus.cc:374] T ec02a14916dc4516aebb49e65c388d59 P b398b7c2ea92406190c76ed6d810a151 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:29.378588 23700 raft_consensus.cc:729] T ec02a14916dc4516aebb49e65c388d59 P b398b7c2ea92406190c76ed6d810a151 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b398b7c2ea92406190c76ed6d810a151, State: Initialized, Role: FOLLOWER
I20260619 18:42:29.378710 23700 consensus_queue.cc:260] T ec02a14916dc4516aebb49e65c388d59 P b398b7c2ea92406190c76ed6d810a151 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } } peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } }
I20260619 18:42:29.378937 23700 ts_tablet_manager.cc:1428] T ec02a14916dc4516aebb49e65c388d59 P b398b7c2ea92406190c76ed6d810a151: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 18:42:29.379281 23179 catalog_manager.cc:4841] TS 67b59204f8214e069ec7c042cae0733c (127.21.143.67:43603): tablet c64a424847d04029a50baf00f6cb1eed (table test-table [id=dbdc82c91fa749758da86101f2076cd5]) successfully deleted
W20260619 18:42:29.424340 23677 tablet.cc:2369] T ec02a14916dc4516aebb49e65c388d59 P 67b59204f8214e069ec7c042cae0733c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:42:29.440397 23706 raft_consensus.cc:482] T ec02a14916dc4516aebb49e65c388d59 P b398b7c2ea92406190c76ed6d810a151 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:42:29.440546 23706 raft_consensus.cc:504] T ec02a14916dc4516aebb49e65c388d59 P b398b7c2ea92406190c76ed6d810a151 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } } peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } }
I20260619 18:42:29.441709 23706 leader_election.cc:288] T ec02a14916dc4516aebb49e65c388d59 P b398b7c2ea92406190c76ed6d810a151 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72b20147812547e08d6ee9cda7644bfa (127.21.143.66:39133), 67b59204f8214e069ec7c042cae0733c (127.21.143.67:43603)
I20260619 18:42:29.447027 23697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:29.442425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:50575 (local address 127.21.143.67:43603)
0619 18:42:29.442536 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:29.442540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:29.442825 (+ 285us) server_negotiation.cc:407] Connection header received
0619 18:42:29.442996 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:29.442999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:29.443054 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:29.443100 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:29.443474 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.444214 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.445599 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.446208 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.446340 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:29.446394 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:29.446828 (+ 434us) server_negotiation.cc:299] Negotiation successful
0619 18:42:29.446910 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 18:42:29.447034 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:29.442316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:38803 (local address 127.21.143.66:39133)
0619 18:42:29.442431 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:29.442435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:29.442658 (+ 223us) server_negotiation.cc:407] Connection header received
0619 18:42:29.442849 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:29.442852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:29.442897 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:29.442941 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:29.443353 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.444120 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.445694 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.446264 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.446361 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:29.446405 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:29.446829 (+ 424us) server_negotiation.cc:299] Negotiation successful
0619 18:42:29.446902 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 18:42:29.447419 23728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:29.442040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:43603 (local address 127.21.143.65:50575)
0619 18:42:29.442752 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:29.442764 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:29.442889 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:29.443222 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:29.443224 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:29.443238 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:29.443364 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:29.443370 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.444342 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.444345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:29.445472 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:29.445481 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.446470 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.446475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:29.446663 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:29.446703 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 18:42:29.447153 (+ 450us) client_negotiation.cc:240] Negotiation successful
0619 18:42:29.447246 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":31,"thread_start_us":105,"threads_started":1}
I20260619 18:42:29.447456 23729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:29.441952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:39133 (local address 127.21.143.65:38803)
0619 18:42:29.442579 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:29.442593 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:29.442734 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:29.443066 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:29.443068 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:29.443082 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:29.443212 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:29.443218 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.444258 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.444261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:29.445569 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:29.445580 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.446481 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.446484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:29.446659 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:29.446708 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 18:42:29.447259 (+ 551us) client_negotiation.cc:240] Negotiation successful
0619 18:42:29.447340 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"mutex_wait_us":142,"thread_start_us":97,"threads_started":1}
I20260619 18:42:29.447860 23630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec02a14916dc4516aebb49e65c388d59" candidate_uuid: "b398b7c2ea92406190c76ed6d810a151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b59204f8214e069ec7c042cae0733c" is_pre_election: true
I20260619 18:42:29.448027 23630 raft_consensus.cc:2454] T ec02a14916dc4516aebb49e65c388d59 P 67b59204f8214e069ec7c042cae0733c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b398b7c2ea92406190c76ed6d810a151 in term 0.
I20260619 18:42:29.448125 23490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec02a14916dc4516aebb49e65c388d59" candidate_uuid: "b398b7c2ea92406190c76ed6d810a151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b20147812547e08d6ee9cda7644bfa" is_pre_election: true
I20260619 18:42:29.448367 23490 raft_consensus.cc:2454] T ec02a14916dc4516aebb49e65c388d59 P 72b20147812547e08d6ee9cda7644bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b398b7c2ea92406190c76ed6d810a151 in term 0.
I20260619 18:42:29.448406 23283 leader_election.cc:302] T ec02a14916dc4516aebb49e65c388d59 P b398b7c2ea92406190c76ed6d810a151 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b59204f8214e069ec7c042cae0733c, b398b7c2ea92406190c76ed6d810a151; no voters:
I20260619 18:42:29.448632 23706 raft_consensus.cc:2781] T ec02a14916dc4516aebb49e65c388d59 P b398b7c2ea92406190c76ed6d810a151 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:42:29.448691 23706 raft_consensus.cc:482] T ec02a14916dc4516aebb49e65c388d59 P b398b7c2ea92406190c76ed6d810a151 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:42:29.448736 23706 raft_consensus.cc:3037] T ec02a14916dc4516aebb49e65c388d59 P b398b7c2ea92406190c76ed6d810a151 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:29.449795 23706 raft_consensus.cc:504] T ec02a14916dc4516aebb49e65c388d59 P b398b7c2ea92406190c76ed6d810a151 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } } peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } }
I20260619 18:42:29.450187 23706 leader_election.cc:288] T ec02a14916dc4516aebb49e65c388d59 P b398b7c2ea92406190c76ed6d810a151 [CANDIDATE]: Term 1 election: Requested vote from peers 72b20147812547e08d6ee9cda7644bfa (127.21.143.66:39133), 67b59204f8214e069ec7c042cae0733c (127.21.143.67:43603)
I20260619 18:42:29.450438 23490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec02a14916dc4516aebb49e65c388d59" candidate_uuid: "b398b7c2ea92406190c76ed6d810a151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b20147812547e08d6ee9cda7644bfa"
I20260619 18:42:29.450519 23630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec02a14916dc4516aebb49e65c388d59" candidate_uuid: "b398b7c2ea92406190c76ed6d810a151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b59204f8214e069ec7c042cae0733c"
I20260619 18:42:29.450577 23490 raft_consensus.cc:3037] T ec02a14916dc4516aebb49e65c388d59 P 72b20147812547e08d6ee9cda7644bfa [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:29.450637 23630 raft_consensus.cc:3037] T ec02a14916dc4516aebb49e65c388d59 P 67b59204f8214e069ec7c042cae0733c [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:29.451529 23490 raft_consensus.cc:2454] T ec02a14916dc4516aebb49e65c388d59 P 72b20147812547e08d6ee9cda7644bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b398b7c2ea92406190c76ed6d810a151 in term 1.
I20260619 18:42:29.451799 23282 leader_election.cc:302] T ec02a14916dc4516aebb49e65c388d59 P b398b7c2ea92406190c76ed6d810a151 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72b20147812547e08d6ee9cda7644bfa, b398b7c2ea92406190c76ed6d810a151; no voters:
I20260619 18:42:29.451803 23630 raft_consensus.cc:2454] T ec02a14916dc4516aebb49e65c388d59 P 67b59204f8214e069ec7c042cae0733c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b398b7c2ea92406190c76ed6d810a151 in term 1.
I20260619 18:42:29.451978 23706 raft_consensus.cc:2781] T ec02a14916dc4516aebb49e65c388d59 P b398b7c2ea92406190c76ed6d810a151 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:29.452199 23706 raft_consensus.cc:686] T ec02a14916dc4516aebb49e65c388d59 P b398b7c2ea92406190c76ed6d810a151 [term 1 LEADER]: Becoming Leader. State: Replica: b398b7c2ea92406190c76ed6d810a151, State: Running, Role: LEADER
I20260619 18:42:29.452436 23706 consensus_queue.cc:237] T ec02a14916dc4516aebb49e65c388d59 P b398b7c2ea92406190c76ed6d810a151 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } } peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } }
W20260619 18:42:29.454583 23537 tablet.cc:2369] T ec02a14916dc4516aebb49e65c388d59 P 72b20147812547e08d6ee9cda7644bfa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:42:29.454859 23193 catalog_manager.cc:5495] T ec02a14916dc4516aebb49e65c388d59 P b398b7c2ea92406190c76ed6d810a151 reported cstate change: term changed from 0 to 1, leader changed from <none> to b398b7c2ea92406190c76ed6d810a151 (127.21.143.65). New cstate: current_term: 1 leader_uuid: "b398b7c2ea92406190c76ed6d810a151" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } health_report { overall_health: UNKNOWN } } }
Jun 19 18:42:29 dist-test-slave-mld5 krb5kdc[23148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894548, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:29.493981 23737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:29.484958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:45809 (local address 127.0.0.1:48824)
0619 18:42:29.485305 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:29.485317 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:29.485411 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:29.485650 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:29.485651 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:29.486020 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:29.486114 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:29.486120 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.487152 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.487155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:29.487808 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:29.487817 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.488389 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.488392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:29.488477 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:29.489080 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:29.489091 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:29.490461 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:29.492401 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:29.492408 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:29.492414 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:29.492642 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:29.492935 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:29.492937 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:29.492938 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:29.493020 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:29.493466 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:29.493471 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:29.493577 (+ 106us) client_negotiation.cc:769] Sending connection context
0619 18:42:29.493670 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 18:42:29.493847 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":2,"thread_start_us":117,"threads_started":1}
I20260619 18:42:29.494105 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:29.485034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48824 (local address 127.21.143.126:45809)
0619 18:42:29.485166 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:29.485171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:29.485401 (+ 230us) server_negotiation.cc:407] Connection header received
0619 18:42:29.485483 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:29.485486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:29.485537 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:29.485645 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:29.486243 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.487020 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.487935 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.488274 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.488392 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:29.490609 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:29.490613 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:29.490616 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:29.490640 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:29.492279 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:29.492745 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:29.492748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:29.492750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:29.492841 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:29.493115 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:29.493119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:29.493121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:29.493353 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:29.493461 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:29.493764 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 18:42:29.493925 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":54}
W20260619 18:42:29.495477 23397 tablet.cc:2369] T ec02a14916dc4516aebb49e65c388d59 P b398b7c2ea92406190c76ed6d810a151: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:42:29.498485 23193 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:48824:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 18:42:29.499949 23193 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:42:29.504714 23330 tablet_service.cc:1467] Processing CreateTablet for tablet d653f30838014f979fbb804fd609c00b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=55894df59d9944368162a52ab02b5558]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:42:29.504930 23470 tablet_service.cc:1467] Processing CreateTablet for tablet d653f30838014f979fbb804fd609c00b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=55894df59d9944368162a52ab02b5558]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:42:29.505126 23330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d653f30838014f979fbb804fd609c00b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:29.505100 23610 tablet_service.cc:1467] Processing CreateTablet for tablet d653f30838014f979fbb804fd609c00b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=55894df59d9944368162a52ab02b5558]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:42:29.505324 23470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d653f30838014f979fbb804fd609c00b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:29.505448 23610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d653f30838014f979fbb804fd609c00b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:29.507584 23701 tablet_bootstrap.cc:492] T d653f30838014f979fbb804fd609c00b P 72b20147812547e08d6ee9cda7644bfa: Bootstrap starting.
I20260619 18:42:29.508162 23699 tablet_bootstrap.cc:492] T d653f30838014f979fbb804fd609c00b P 67b59204f8214e069ec7c042cae0733c: Bootstrap starting.
I20260619 18:42:29.508160 23700 tablet_bootstrap.cc:492] T d653f30838014f979fbb804fd609c00b P b398b7c2ea92406190c76ed6d810a151: Bootstrap starting.
I20260619 18:42:29.508884 23701 tablet_bootstrap.cc:654] T d653f30838014f979fbb804fd609c00b P 72b20147812547e08d6ee9cda7644bfa: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:29.509132 23699 tablet_bootstrap.cc:654] T d653f30838014f979fbb804fd609c00b P 67b59204f8214e069ec7c042cae0733c: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:29.509132 23700 tablet_bootstrap.cc:654] T d653f30838014f979fbb804fd609c00b P b398b7c2ea92406190c76ed6d810a151: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:29.510469 23699 tablet_bootstrap.cc:492] T d653f30838014f979fbb804fd609c00b P 67b59204f8214e069ec7c042cae0733c: No bootstrap required, opened a new log
I20260619 18:42:29.510560 23699 ts_tablet_manager.cc:1397] T d653f30838014f979fbb804fd609c00b P 67b59204f8214e069ec7c042cae0733c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 18:42:29.510777 23700 tablet_bootstrap.cc:492] T d653f30838014f979fbb804fd609c00b P b398b7c2ea92406190c76ed6d810a151: No bootstrap required, opened a new log
I20260619 18:42:29.510859 23700 ts_tablet_manager.cc:1397] T d653f30838014f979fbb804fd609c00b P b398b7c2ea92406190c76ed6d810a151: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 18:42:29.511008 23699 raft_consensus.cc:348] T d653f30838014f979fbb804fd609c00b P 67b59204f8214e069ec7c042cae0733c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } } peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } }
I20260619 18:42:29.511133 23699 raft_consensus.cc:374] T d653f30838014f979fbb804fd609c00b P 67b59204f8214e069ec7c042cae0733c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:29.511174 23699 raft_consensus.cc:729] T d653f30838014f979fbb804fd609c00b P 67b59204f8214e069ec7c042cae0733c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b59204f8214e069ec7c042cae0733c, State: Initialized, Role: FOLLOWER
I20260619 18:42:29.511317 23699 consensus_queue.cc:260] T d653f30838014f979fbb804fd609c00b P 67b59204f8214e069ec7c042cae0733c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } } peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } }
I20260619 18:42:29.511390 23700 raft_consensus.cc:348] T d653f30838014f979fbb804fd609c00b P b398b7c2ea92406190c76ed6d810a151 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } } peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } }
I20260619 18:42:29.511493 23700 raft_consensus.cc:374] T d653f30838014f979fbb804fd609c00b P b398b7c2ea92406190c76ed6d810a151 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:29.511533 23700 raft_consensus.cc:729] T d653f30838014f979fbb804fd609c00b P b398b7c2ea92406190c76ed6d810a151 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b398b7c2ea92406190c76ed6d810a151, State: Initialized, Role: FOLLOWER
I20260619 18:42:29.511615 23699 ts_tablet_manager.cc:1428] T d653f30838014f979fbb804fd609c00b P 67b59204f8214e069ec7c042cae0733c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:42:29.511657 23700 consensus_queue.cc:260] T d653f30838014f979fbb804fd609c00b P b398b7c2ea92406190c76ed6d810a151 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } } peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } }
I20260619 18:42:29.511816 23705 tablet_replica.cc:439] T d653f30838014f979fbb804fd609c00b P 67b59204f8214e069ec7c042cae0733c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } } peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } } }
I20260619 18:42:29.511924 23700 ts_tablet_manager.cc:1428] T d653f30838014f979fbb804fd609c00b P b398b7c2ea92406190c76ed6d810a151: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:42:29.511950 23705 tablet_replica.cc:442] T d653f30838014f979fbb804fd609c00b P 67b59204f8214e069ec7c042cae0733c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:42:29.512110 23706 tablet_replica.cc:439] T d653f30838014f979fbb804fd609c00b P b398b7c2ea92406190c76ed6d810a151: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } } peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } } }
I20260619 18:42:29.512229 23706 tablet_replica.cc:442] T d653f30838014f979fbb804fd609c00b P b398b7c2ea92406190c76ed6d810a151: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:42:29.512284 23701 tablet_bootstrap.cc:492] T d653f30838014f979fbb804fd609c00b P 72b20147812547e08d6ee9cda7644bfa: No bootstrap required, opened a new log
I20260619 18:42:29.512354 23701 ts_tablet_manager.cc:1397] T d653f30838014f979fbb804fd609c00b P 72b20147812547e08d6ee9cda7644bfa: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 18:42:29.512745 23701 raft_consensus.cc:348] T d653f30838014f979fbb804fd609c00b P 72b20147812547e08d6ee9cda7644bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } } peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } }
I20260619 18:42:29.512844 23701 raft_consensus.cc:374] T d653f30838014f979fbb804fd609c00b P 72b20147812547e08d6ee9cda7644bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:29.512872 23701 raft_consensus.cc:729] T d653f30838014f979fbb804fd609c00b P 72b20147812547e08d6ee9cda7644bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72b20147812547e08d6ee9cda7644bfa, State: Initialized, Role: FOLLOWER
I20260619 18:42:29.513005 23701 consensus_queue.cc:260] T d653f30838014f979fbb804fd609c00b P 72b20147812547e08d6ee9cda7644bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } } peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } }
I20260619 18:42:29.513266 23701 ts_tablet_manager.cc:1428] T d653f30838014f979fbb804fd609c00b P 72b20147812547e08d6ee9cda7644bfa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:42:29.513216 23717 tablet_replica.cc:439] T d653f30838014f979fbb804fd609c00b P 72b20147812547e08d6ee9cda7644bfa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } } peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } } }
I20260619 18:42:29.513325 23717 tablet_replica.cc:442] T d653f30838014f979fbb804fd609c00b P 72b20147812547e08d6ee9cda7644bfa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:42:29.544378 23706 raft_consensus.cc:482] T d653f30838014f979fbb804fd609c00b P b398b7c2ea92406190c76ed6d810a151 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:42:29.544515 23706 raft_consensus.cc:504] T d653f30838014f979fbb804fd609c00b P b398b7c2ea92406190c76ed6d810a151 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } } peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } }
I20260619 18:42:29.545168 23706 leader_election.cc:288] T d653f30838014f979fbb804fd609c00b P b398b7c2ea92406190c76ed6d810a151 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72b20147812547e08d6ee9cda7644bfa (127.21.143.66:39133), 67b59204f8214e069ec7c042cae0733c (127.21.143.67:43603)
I20260619 18:42:29.545558 23490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653f30838014f979fbb804fd609c00b" candidate_uuid: "b398b7c2ea92406190c76ed6d810a151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b20147812547e08d6ee9cda7644bfa" is_pre_election: true
I20260619 18:42:29.545562 23630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653f30838014f979fbb804fd609c00b" candidate_uuid: "b398b7c2ea92406190c76ed6d810a151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b59204f8214e069ec7c042cae0733c" is_pre_election: true
I20260619 18:42:29.545719 23490 raft_consensus.cc:2454] T d653f30838014f979fbb804fd609c00b P 72b20147812547e08d6ee9cda7644bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b398b7c2ea92406190c76ed6d810a151 in term 0.
I20260619 18:42:29.545732 23630 raft_consensus.cc:2454] T d653f30838014f979fbb804fd609c00b P 67b59204f8214e069ec7c042cae0733c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b398b7c2ea92406190c76ed6d810a151 in term 0.
I20260619 18:42:29.546031 23282 leader_election.cc:302] T d653f30838014f979fbb804fd609c00b P b398b7c2ea92406190c76ed6d810a151 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72b20147812547e08d6ee9cda7644bfa, b398b7c2ea92406190c76ed6d810a151; no voters:
I20260619 18:42:29.546217 23706 raft_consensus.cc:2781] T d653f30838014f979fbb804fd609c00b P b398b7c2ea92406190c76ed6d810a151 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:42:29.546279 23706 raft_consensus.cc:482] T d653f30838014f979fbb804fd609c00b P b398b7c2ea92406190c76ed6d810a151 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:42:29.546308 23706 raft_consensus.cc:3037] T d653f30838014f979fbb804fd609c00b P b398b7c2ea92406190c76ed6d810a151 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:29.547371 23706 raft_consensus.cc:504] T d653f30838014f979fbb804fd609c00b P b398b7c2ea92406190c76ed6d810a151 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } } peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } }
I20260619 18:42:29.547745 23706 leader_election.cc:288] T d653f30838014f979fbb804fd609c00b P b398b7c2ea92406190c76ed6d810a151 [CANDIDATE]: Term 1 election: Requested vote from peers 72b20147812547e08d6ee9cda7644bfa (127.21.143.66:39133), 67b59204f8214e069ec7c042cae0733c (127.21.143.67:43603)
I20260619 18:42:29.548044 23490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653f30838014f979fbb804fd609c00b" candidate_uuid: "b398b7c2ea92406190c76ed6d810a151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b20147812547e08d6ee9cda7644bfa"
I20260619 18:42:29.548110 23630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653f30838014f979fbb804fd609c00b" candidate_uuid: "b398b7c2ea92406190c76ed6d810a151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b59204f8214e069ec7c042cae0733c"
I20260619 18:42:29.548182 23490 raft_consensus.cc:3037] T d653f30838014f979fbb804fd609c00b P 72b20147812547e08d6ee9cda7644bfa [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:29.548240 23630 raft_consensus.cc:3037] T d653f30838014f979fbb804fd609c00b P 67b59204f8214e069ec7c042cae0733c [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:29.549273 23490 raft_consensus.cc:2454] T d653f30838014f979fbb804fd609c00b P 72b20147812547e08d6ee9cda7644bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b398b7c2ea92406190c76ed6d810a151 in term 1.
I20260619 18:42:29.549336 23630 raft_consensus.cc:2454] T d653f30838014f979fbb804fd609c00b P 67b59204f8214e069ec7c042cae0733c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b398b7c2ea92406190c76ed6d810a151 in term 1.
I20260619 18:42:29.549515 23282 leader_election.cc:302] T d653f30838014f979fbb804fd609c00b P b398b7c2ea92406190c76ed6d810a151 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72b20147812547e08d6ee9cda7644bfa, b398b7c2ea92406190c76ed6d810a151; no voters:
I20260619 18:42:29.549681 23706 raft_consensus.cc:2781] T d653f30838014f979fbb804fd609c00b P b398b7c2ea92406190c76ed6d810a151 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:29.549780 23706 raft_consensus.cc:686] T d653f30838014f979fbb804fd609c00b P b398b7c2ea92406190c76ed6d810a151 [term 1 LEADER]: Becoming Leader. State: Replica: b398b7c2ea92406190c76ed6d810a151, State: Running, Role: LEADER
I20260619 18:42:29.549912 23706 consensus_queue.cc:237] T d653f30838014f979fbb804fd609c00b P b398b7c2ea92406190c76ed6d810a151 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } } peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } }
I20260619 18:42:29.550550 23730 tablet_replica.cc:439] T d653f30838014f979fbb804fd609c00b P b398b7c2ea92406190c76ed6d810a151: TxnStatusTablet state changed. Reason: New leader b398b7c2ea92406190c76ed6d810a151. Latest consensus state: current_term: 1 leader_uuid: "b398b7c2ea92406190c76ed6d810a151" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } } peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } } }
I20260619 18:42:29.550654 23730 tablet_replica.cc:442] T d653f30838014f979fbb804fd609c00b P b398b7c2ea92406190c76ed6d810a151: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:42:29.551084 23742 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 18:42:29.551265 23742 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 18:42:29.551632 23192 catalog_manager.cc:5495] T d653f30838014f979fbb804fd609c00b P b398b7c2ea92406190c76ed6d810a151 reported cstate change: term changed from 0 to 1, leader changed from <none> to b398b7c2ea92406190c76ed6d810a151 (127.21.143.65). New cstate: current_term: 1 leader_uuid: "b398b7c2ea92406190c76ed6d810a151" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } health_report { overall_health: UNKNOWN } } }
I20260619 18:42:29.567572 23698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:29.564111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51646 (local address 127.21.143.65:43707)
0619 18:42:29.564238 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:29.564242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:29.564256 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:42:29.564405 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:29.564410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:29.564464 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:29.564595 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:42:29.564837 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.565344 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.566187 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.566395 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.566498 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:29.566767 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:42:29.567152 (+ 385us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:42:29.567287 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:29.567385 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 18:42:29.567437 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 18:42:29.567572 23737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:29.563953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:43707 (local address 127.0.0.1:51646)
0619 18:42:29.564109 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:29.564132 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:29.564259 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:29.564622 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:29.564625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:29.564632 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:42:29.564735 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:29.564740 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.565451 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.565456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:29.566074 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:29.566081 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.566496 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.566500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:29.566589 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:29.566627 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:42:29.567262 (+ 635us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:42:29.567277 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 18:42:29.567377 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 18:42:29.567450 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 18:42:29.570216 23630 raft_consensus.cc:1261] T d653f30838014f979fbb804fd609c00b P 67b59204f8214e069ec7c042cae0733c [term 1 FOLLOWER]: Refusing update from remote peer b398b7c2ea92406190c76ed6d810a151: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:42:29.570433 23490 raft_consensus.cc:1261] T d653f30838014f979fbb804fd609c00b P 72b20147812547e08d6ee9cda7644bfa [term 1 FOLLOWER]: Refusing update from remote peer b398b7c2ea92406190c76ed6d810a151: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:42:29.570995 23706 consensus_queue.cc:1035] T d653f30838014f979fbb804fd609c00b P b398b7c2ea92406190c76ed6d810a151 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:42:29.571365 23706 consensus_queue.cc:1035] T d653f30838014f979fbb804fd609c00b P b398b7c2ea92406190c76ed6d810a151 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:42:29.572669 23717 tablet_replica.cc:439] T d653f30838014f979fbb804fd609c00b P 72b20147812547e08d6ee9cda7644bfa: TxnStatusTablet state changed. Reason: New leader b398b7c2ea92406190c76ed6d810a151. Latest consensus state: current_term: 1 leader_uuid: "b398b7c2ea92406190c76ed6d810a151" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } } peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } } }
I20260619 18:42:29.572772 23717 tablet_replica.cc:442] T d653f30838014f979fbb804fd609c00b P 72b20147812547e08d6ee9cda7644bfa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:42:29.572949 23705 tablet_replica.cc:439] T d653f30838014f979fbb804fd609c00b P 67b59204f8214e069ec7c042cae0733c: TxnStatusTablet state changed. Reason: New leader b398b7c2ea92406190c76ed6d810a151. Latest consensus state: current_term: 1 leader_uuid: "b398b7c2ea92406190c76ed6d810a151" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } } peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } } }
I20260619 18:42:29.573064 23705 tablet_replica.cc:442] T d653f30838014f979fbb804fd609c00b P 67b59204f8214e069ec7c042cae0733c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:42:29.574306 23706 tablet_replica.cc:439] T d653f30838014f979fbb804fd609c00b P b398b7c2ea92406190c76ed6d810a151: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b398b7c2ea92406190c76ed6d810a151" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } } peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } } }
I20260619 18:42:29.574424 23706 tablet_replica.cc:442] T d653f30838014f979fbb804fd609c00b P b398b7c2ea92406190c76ed6d810a151: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:42:29.575738 23706 tablet_replica.cc:439] T d653f30838014f979fbb804fd609c00b P b398b7c2ea92406190c76ed6d810a151: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b398b7c2ea92406190c76ed6d810a151" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } } peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } } }
I20260619 18:42:29.575830 23706 tablet_replica.cc:442] T d653f30838014f979fbb804fd609c00b P b398b7c2ea92406190c76ed6d810a151: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:42:29.576515 23705 tablet_replica.cc:439] T d653f30838014f979fbb804fd609c00b P 67b59204f8214e069ec7c042cae0733c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b398b7c2ea92406190c76ed6d810a151" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } } peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } } }
I20260619 18:42:29.576611 23705 tablet_replica.cc:442] T d653f30838014f979fbb804fd609c00b P 67b59204f8214e069ec7c042cae0733c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:42:29.577324 23717 tablet_replica.cc:439] T d653f30838014f979fbb804fd609c00b P 72b20147812547e08d6ee9cda7644bfa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b398b7c2ea92406190c76ed6d810a151" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b398b7c2ea92406190c76ed6d810a151" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 43707 } } peers { permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 } } peers { permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 } } }
I20260619 18:42:29.577437 23717 tablet_replica.cc:442] T d653f30838014f979fbb804fd609c00b P 72b20147812547e08d6ee9cda7644bfa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:42:29.583549 23698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:29.580291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51660 (local address 127.21.143.65:43707)
0619 18:42:29.580387 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:29.580391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:29.580429 (+ 38us) server_negotiation.cc:407] Connection header received
0619 18:42:29.580558 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:29.580560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:29.580595 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:29.580654 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:42:29.580898 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.581374 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.582495 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.582699 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.582811 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:29.583068 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:42:29.583129 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:42:29.583182 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:29.583339 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 18:42:29.583389 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 18:42:29.594046 23698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:29.590401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:49713 (local address 127.21.143.65:43707)
0619 18:42:29.590515 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:29.590519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:29.590748 (+ 229us) server_negotiation.cc:407] Connection header received
0619 18:42:29.590898 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:29.590901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:29.590940 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:29.591012 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:42:29.591293 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.591836 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.593046 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.593244 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.593322 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:29.593622 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:42:29.593676 (+ 54us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:42:29.593723 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:29.593859 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 18:42:29.593900 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 18:42:29.594089 23749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:29.590301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:43707 (local address 127.21.143.65:49713)
0619 18:42:29.590656 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:29.590669 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:29.590774 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:29.591038 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:29.591041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:29.591048 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:42:29.591168 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:29.591175 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.591942 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.591945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:29.592928 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:29.592939 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.593350 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.593353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:29.593448 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:29.593495 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:42:29.593769 (+ 274us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:42:29.593775 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 18:42:29.593840 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 18:42:29.593913 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":95,"threads_started":1}
I20260619 18:42:29.600489 23630 raft_consensus.cc:1261] T ec02a14916dc4516aebb49e65c388d59 P 67b59204f8214e069ec7c042cae0733c [term 1 FOLLOWER]: Refusing update from remote peer b398b7c2ea92406190c76ed6d810a151: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:42:29.600569 23490 raft_consensus.cc:1261] T ec02a14916dc4516aebb49e65c388d59 P 72b20147812547e08d6ee9cda7644bfa [term 1 FOLLOWER]: Refusing update from remote peer b398b7c2ea92406190c76ed6d810a151: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:42:29.600849 23706 consensus_queue.cc:1035] T ec02a14916dc4516aebb49e65c388d59 P b398b7c2ea92406190c76ed6d810a151 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b59204f8214e069ec7c042cae0733c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:42:29.601116 23730 consensus_queue.cc:1035] T ec02a14916dc4516aebb49e65c388d59 P b398b7c2ea92406190c76ed6d810a151 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b20147812547e08d6ee9cda7644bfa" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:42:29.605993 23712 mvcc.cc:205] Tried to move back new op lower bound from 7298640075160350720 to 7298640074559422464. Current Snapshot: MvccSnapshot[applied={T|T < 7298640075160350720}]
I20260619 18:42:29.637665 23191 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:42:29.637856 23191 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:42:29.639832 23191 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed2f4a5d94cd421eb4f87baf37b71afa: Sending DeleteTablet for 3 replicas of tablet ec02a14916dc4516aebb49e65c388d59
I20260619 18:42:29.640362 23470 tablet_service.cc:1514] Processing DeleteTablet for tablet ec02a14916dc4516aebb49e65c388d59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:42:29 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:51210
I20260619 18:42:29.640416 23610 tablet_service.cc:1514] Processing DeleteTablet for tablet ec02a14916dc4516aebb49e65c388d59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:42:29 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:48364
I20260619 18:42:29.640520 23723 tablet_replica.cc:330] T ec02a14916dc4516aebb49e65c388d59 P 72b20147812547e08d6ee9cda7644bfa: stopping tablet replica
I20260619 18:42:29.640561 23722 tablet_replica.cc:330] T ec02a14916dc4516aebb49e65c388d59 P 67b59204f8214e069ec7c042cae0733c: stopping tablet replica
I20260619 18:42:29.640637 23723 raft_consensus.cc:2229] T ec02a14916dc4516aebb49e65c388d59 P 72b20147812547e08d6ee9cda7644bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:42:29.640650 23722 raft_consensus.cc:2229] T ec02a14916dc4516aebb49e65c388d59 P 67b59204f8214e069ec7c042cae0733c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:42:29.640748 23723 raft_consensus.cc:2258] T ec02a14916dc4516aebb49e65c388d59 P 72b20147812547e08d6ee9cda7644bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:42:29.640765 23722 raft_consensus.cc:2258] T ec02a14916dc4516aebb49e65c388d59 P 67b59204f8214e069ec7c042cae0733c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:42:29.641331 23329 tablet_service.cc:1514] Processing DeleteTablet for tablet ec02a14916dc4516aebb49e65c388d59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:42:29 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:51636
I20260619 18:42:29.641424 23722 ts_tablet_manager.cc:1905] T ec02a14916dc4516aebb49e65c388d59 P 67b59204f8214e069ec7c042cae0733c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:42:29.641528 23724 tablet_replica.cc:330] T ec02a14916dc4516aebb49e65c388d59 P b398b7c2ea92406190c76ed6d810a151: stopping tablet replica
I20260619 18:42:29.641647 23724 raft_consensus.cc:2229] T ec02a14916dc4516aebb49e65c388d59 P b398b7c2ea92406190c76ed6d810a151 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:42:29.641913 23724 raft_consensus.cc:2258] T ec02a14916dc4516aebb49e65c388d59 P b398b7c2ea92406190c76ed6d810a151 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:42:29.642046 23723 ts_tablet_manager.cc:1905] T ec02a14916dc4516aebb49e65c388d59 P 72b20147812547e08d6ee9cda7644bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:42:29.642628 23724 ts_tablet_manager.cc:1905] T ec02a14916dc4516aebb49e65c388d59 P b398b7c2ea92406190c76ed6d810a151: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:42:29.644114 23722 ts_tablet_manager.cc:1918] T ec02a14916dc4516aebb49e65c388d59 P 67b59204f8214e069ec7c042cae0733c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:42:29.644179 23722 log.cc:1196] T ec02a14916dc4516aebb49e65c388d59 P 67b59204f8214e069ec7c042cae0733c: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-2/wal/wals/ec02a14916dc4516aebb49e65c388d59
I20260619 18:42:29.644428 23722 ts_tablet_manager.cc:1939] T ec02a14916dc4516aebb49e65c388d59 P 67b59204f8214e069ec7c042cae0733c: Deleting consensus metadata
I20260619 18:42:29.644752 23723 ts_tablet_manager.cc:1918] T ec02a14916dc4516aebb49e65c388d59 P 72b20147812547e08d6ee9cda7644bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:42:29.644812 23723 log.cc:1196] T ec02a14916dc4516aebb49e65c388d59 P 72b20147812547e08d6ee9cda7644bfa: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-1/wal/wals/ec02a14916dc4516aebb49e65c388d59
I20260619 18:42:29.645046 23723 ts_tablet_manager.cc:1939] T ec02a14916dc4516aebb49e65c388d59 P 72b20147812547e08d6ee9cda7644bfa: Deleting consensus metadata
I20260619 18:42:29.645109 23179 catalog_manager.cc:4841] TS 67b59204f8214e069ec7c042cae0733c (127.21.143.67:43603): tablet ec02a14916dc4516aebb49e65c388d59 (table test-table [id=072f84c9054445e3b3bd7e87a23ab6b7]) successfully deleted
I20260619 18:42:29.645438 23724 ts_tablet_manager.cc:1918] T ec02a14916dc4516aebb49e65c388d59 P b398b7c2ea92406190c76ed6d810a151: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:42:29.645500 23724 log.cc:1196] T ec02a14916dc4516aebb49e65c388d59 P b398b7c2ea92406190c76ed6d810a151: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894542572984-22077-0/minicluster-data/ts-0/wal/wals/ec02a14916dc4516aebb49e65c388d59
I20260619 18:42:29.645736 23724 ts_tablet_manager.cc:1939] T ec02a14916dc4516aebb49e65c388d59 P b398b7c2ea92406190c76ed6d810a151: Deleting consensus metadata
I20260619 18:42:29.645737 23178 catalog_manager.cc:4841] TS 72b20147812547e08d6ee9cda7644bfa (127.21.143.66:39133): tablet ec02a14916dc4516aebb49e65c388d59 (table test-table [id=072f84c9054445e3b3bd7e87a23ab6b7]) successfully deleted
I20260619 18:42:29.646351 23178 catalog_manager.cc:4841] TS b398b7c2ea92406190c76ed6d810a151 (127.21.143.65:43707): tablet ec02a14916dc4516aebb49e65c388d59 (table test-table [id=072f84c9054445e3b3bd7e87a23ab6b7]) successfully deleted
Jun 19 18:42:29 dist-test-slave-mld5 krb5kdc[23148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894549, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.143.65@KRBTEST.COM
I20260619 18:42:29.655716 23698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:29.643388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51674 (local address 127.21.143.65:43707)
0619 18:42:29.645382 (+ 1994us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:29.645387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:29.646101 (+ 714us) server_negotiation.cc:407] Connection header received
0619 18:42:29.646287 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:29.646290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:29.646350 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:29.646450 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:29.647130 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.647717 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.648615 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.648837 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.648940 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:29.652193 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:29.652201 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:29.652212 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:29.652232 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:29.653952 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:29.654421 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:29.654426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:29.654427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:29.654502 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:29.654820 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:29.654822 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:29.654823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:29.655061 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:29.655209 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:29.655458 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 18:42:29.655553 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1914}
W20260619 18:42:29.656306 23384 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51674
I20260619 18:42:29.665323 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:29.657986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48830 (local address 127.21.143.126:45809)
0619 18:42:29.658111 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:29.658117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:29.658133 (+ 16us) server_negotiation.cc:407] Connection header received
0619 18:42:29.658298 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:29.658300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:29.658353 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:29.658438 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:29.659085 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.659733 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.660552 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:29.660784 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:29.660899 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:29.662119 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:29.662123 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:29.662126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:29.662151 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:29.663659 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:29.664087 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:29.664090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:29.664092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:29.664182 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:29.664445 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:29.664449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:29.664451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:29.664676 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:29.664810 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:29.665043 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 18:42:29.665161 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 18:42:29.665685 23191 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48830
I20260619 18:42:29.666836 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 23265
I20260619 18:42:29.674137 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 23405
I20260619 18:42:29.681133 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 23545
I20260619 18:42:29.686538 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 23161
2026-06-19T18:42:29Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2594 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:42:29 dist-test-slave-mld5 krb5kdc[23760](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:42:29 dist-test-slave-mld5 krb5kdc[23760](info): set up 2 sockets
Jun 19 18:42:29 dist-test-slave-mld5 krb5kdc[23760](info): commencing operation
krb5kdc: starting...
W20260619 18:42:30.723551 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:42:30 dist-test-slave-mld5 krb5kdc[23760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:42:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:42:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.143.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
I20260619 18:42:30.842238 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:45255
--webserver_interface=127.21.143.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:41013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:45255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:30.948740 23773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:30.948992 23773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:30.949033 23773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:30.952939 23773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:42:30.952982 23773 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 18:42:30.952999 23773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:30.953017 23773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:30.953033 23773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:42:30.953047 23773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:42:30.958006 23773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:41013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:45255
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:45255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:30.958957 23773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:30.959884 23773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:30.968317 23779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:30.968330 23778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:30.968313 23781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:30.968410 23773 server_base.cc:1034] running on GCE node
I20260619 18:42:30.969082 23773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:30.970063 23773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:30.971211 23773 hybrid_clock.cc:648] HybridClock initialized: now 1781894550971181 us; error 44 us; skew 500 ppm
Jun 19 18:42:30 dist-test-slave-mld5 krb5kdc[23760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894550, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:30.973961 23773 init.cc:377] Logged in from keytab as kudu/127.21.143.126@KRBTEST.COM (short username kudu)
I20260619 18:42:30.974840 23773 webserver.cc:460] Webserver started at http://127.21.143.126:35141/ using document root <none> and password file <none>
I20260619 18:42:30.975415 23773 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:30.975484 23773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:30.975651 23773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:30.977883 23773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "299c0445f59a43ad86d5be47c7d9a35c"
format_stamp: "Formatted at 2026-06-19 18:42:30 on dist-test-slave-mld5"
server_key: "17d529ae84671fc428dce7d803eab794"
server_key_iv: "56734eb4f150ac198b3e099bd8ffa0ea"
server_key_version: "encryptionkey@0"
I20260619 18:42:30.978408 23773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "299c0445f59a43ad86d5be47c7d9a35c"
format_stamp: "Formatted at 2026-06-19 18:42:30 on dist-test-slave-mld5"
server_key: "17d529ae84671fc428dce7d803eab794"
server_key_iv: "56734eb4f150ac198b3e099bd8ffa0ea"
server_key_version: "encryptionkey@0"
I20260619 18:42:30.982108 23773 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260619 18:42:30.984395 23788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:30.985607 23773 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:42:30.985752 23773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "299c0445f59a43ad86d5be47c7d9a35c"
format_stamp: "Formatted at 2026-06-19 18:42:30 on dist-test-slave-mld5"
server_key: "17d529ae84671fc428dce7d803eab794"
server_key_iv: "56734eb4f150ac198b3e099bd8ffa0ea"
server_key_version: "encryptionkey@0"
I20260619 18:42:30.985855 23773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:31.004048 23773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:31.007356 23773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:31.007570 23773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:31.015595 23773 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:45255
I20260619 18:42:31.015631 23850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:45255 every 8 connection(s)
I20260619 18:42:31.016542 23773 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:42:31.017963 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 23773
I20260619 18:42:31.018052 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:42:31.018329 22077 external_mini_cluster.cc:1428] Setting key 3dff0384ae4d35ee02f6cdf229c09dbe
I20260619 18:42:31.019644 23851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:31.026544 23851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 299c0445f59a43ad86d5be47c7d9a35c: Bootstrap starting.
Jun 19 18:42:31 dist-test-slave-mld5 krb5kdc[23760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:31.029238 23851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 299c0445f59a43ad86d5be47c7d9a35c: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:31.030246 23851 log.cc:824] T 00000000000000000000000000000000 P 299c0445f59a43ad86d5be47c7d9a35c: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:31.032637 23851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 299c0445f59a43ad86d5be47c7d9a35c: No bootstrap required, opened a new log
I20260619 18:42:31.034268 23854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:31.019642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59080 (local address 127.21.143.126:45255)
0619 18:42:31.020084 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:31.020102 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:31.020155 (+ 53us) server_negotiation.cc:407] Connection header received
0619 18:42:31.020961 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:31.020982 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:31.021434 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:31.021781 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:31.022452 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.023961 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.024911 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.025639 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.025797 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:31.028479 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:31.028486 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:31.028502 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:31.028536 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:31.031084 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:31.031592 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:31.031598 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:31.031603 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:31.031733 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:31.032021 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:31.032024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:31.032025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:31.032528 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:31.032716 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:31.033037 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 18:42:31.033344 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":112,"threads_started":1}
I20260619 18:42:31.036419 23851 raft_consensus.cc:348] T 00000000000000000000000000000000 P 299c0445f59a43ad86d5be47c7d9a35c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "299c0445f59a43ad86d5be47c7d9a35c" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 45255 } }
I20260619 18:42:31.036697 23851 raft_consensus.cc:374] T 00000000000000000000000000000000 P 299c0445f59a43ad86d5be47c7d9a35c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:31.036767 23851 raft_consensus.cc:729] T 00000000000000000000000000000000 P 299c0445f59a43ad86d5be47c7d9a35c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 299c0445f59a43ad86d5be47c7d9a35c, State: Initialized, Role: FOLLOWER
I20260619 18:42:31.037161 23851 consensus_queue.cc:260] T 00000000000000000000000000000000 P 299c0445f59a43ad86d5be47c7d9a35c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "299c0445f59a43ad86d5be47c7d9a35c" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 45255 } }
I20260619 18:42:31.037277 23851 raft_consensus.cc:388] T 00000000000000000000000000000000 P 299c0445f59a43ad86d5be47c7d9a35c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:42:31.037360 23851 raft_consensus.cc:482] T 00000000000000000000000000000000 P 299c0445f59a43ad86d5be47c7d9a35c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:42:31.037452 23851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 299c0445f59a43ad86d5be47c7d9a35c [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:31.038337 23851 raft_consensus.cc:504] T 00000000000000000000000000000000 P 299c0445f59a43ad86d5be47c7d9a35c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "299c0445f59a43ad86d5be47c7d9a35c" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 45255 } }
I20260619 18:42:31.038671 23851 leader_election.cc:302] T 00000000000000000000000000000000 P 299c0445f59a43ad86d5be47c7d9a35c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 299c0445f59a43ad86d5be47c7d9a35c; no voters:
I20260619 18:42:31.039016 23851 leader_election.cc:288] T 00000000000000000000000000000000 P 299c0445f59a43ad86d5be47c7d9a35c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:42:31.039109 23856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 299c0445f59a43ad86d5be47c7d9a35c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:31.039292 23856 raft_consensus.cc:686] T 00000000000000000000000000000000 P 299c0445f59a43ad86d5be47c7d9a35c [term 1 LEADER]: Becoming Leader. State: Replica: 299c0445f59a43ad86d5be47c7d9a35c, State: Running, Role: LEADER
I20260619 18:42:31.039602 23856 consensus_queue.cc:237] T 00000000000000000000000000000000 P 299c0445f59a43ad86d5be47c7d9a35c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "299c0445f59a43ad86d5be47c7d9a35c" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 45255 } }
I20260619 18:42:31.040302 23851 sys_catalog.cc:564] T 00000000000000000000000000000000 P 299c0445f59a43ad86d5be47c7d9a35c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:42:31.041534 23857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 299c0445f59a43ad86d5be47c7d9a35c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "299c0445f59a43ad86d5be47c7d9a35c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "299c0445f59a43ad86d5be47c7d9a35c" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 45255 } } }
I20260619 18:42:31.041574 23858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 299c0445f59a43ad86d5be47c7d9a35c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 299c0445f59a43ad86d5be47c7d9a35c. Latest consensus state: current_term: 1 leader_uuid: "299c0445f59a43ad86d5be47c7d9a35c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "299c0445f59a43ad86d5be47c7d9a35c" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 45255 } } }
I20260619 18:42:31.041651 23857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 299c0445f59a43ad86d5be47c7d9a35c [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:31.041679 23858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 299c0445f59a43ad86d5be47c7d9a35c [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:31.042014 23865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:42:31.045164 23865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:42:31.050551 23865 catalog_manager.cc:1347] Generated new cluster ID: 0615c0ddb5d34e24a942df3d91777dac
I20260619 18:42:31.050609 23865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:42:31.064136 23865 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:42:31.065176 23865 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:42:31.072926 23865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 299c0445f59a43ad86d5be47c7d9a35c: Generated new TSK 0
I20260619 18:42:31.073923 23865 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.143.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab.
Entry for principal kudu/127.21.143.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab.
I20260619 18:42:31.112329 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.65:0
--local_ip_for_outbound_sockets=127.21.143.65
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:45255
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:41013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:31.225937 23877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:31.226243 23877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:31.226353 23877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:31.230609 23877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:31.230692 23877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:31.230759 23877 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 18:42:31.230803 23877 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:42:31.230875 23877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.65
I20260619 18:42:31.236148 23877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:41013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.143.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.143.126:45255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.65
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:31.237248 23877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:31.238372 23877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:31.248030 23883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:31.248031 23882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:31.248065 23877 server_base.cc:1034] running on GCE node
W20260619 18:42:31.248247 23885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:31.248652 23877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:31.249286 23877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:31.250468 23877 hybrid_clock.cc:648] HybridClock initialized: now 1781894551250426 us; error 56 us; skew 500 ppm
Jun 19 18:42:31 dist-test-slave-mld5 krb5kdc[23760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894551, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:31.253172 23877 init.cc:377] Logged in from keytab as kudu/127.21.143.65@KRBTEST.COM (short username kudu)
I20260619 18:42:31.254072 23877 webserver.cc:460] Webserver started at http://127.21.143.65:42025/ using document root <none> and password file <none>
I20260619 18:42:31.254631 23877 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:31.254719 23877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:31.254953 23877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:31.256991 23877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-0/data/instance:
uuid: "217ce9884be4472583704a5d3c829f4e"
format_stamp: "Formatted at 2026-06-19 18:42:31 on dist-test-slave-mld5"
server_key: "8c51e70c7f91794ef8b34089e5d247dd"
server_key_iv: "af08783d0e5cbcb4d7d7d87f6a2b8cdf"
server_key_version: "encryptionkey@0"
I20260619 18:42:31.257542 23877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance:
uuid: "217ce9884be4472583704a5d3c829f4e"
format_stamp: "Formatted at 2026-06-19 18:42:31 on dist-test-slave-mld5"
server_key: "8c51e70c7f91794ef8b34089e5d247dd"
server_key_iv: "af08783d0e5cbcb4d7d7d87f6a2b8cdf"
server_key_version: "encryptionkey@0"
I20260619 18:42:31.261242 23877 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 18:42:31.263571 23892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:31.264818 23877 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 18:42:31.264958 23877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-0/wal
uuid: "217ce9884be4472583704a5d3c829f4e"
format_stamp: "Formatted at 2026-06-19 18:42:31 on dist-test-slave-mld5"
server_key: "8c51e70c7f91794ef8b34089e5d247dd"
server_key_iv: "af08783d0e5cbcb4d7d7d87f6a2b8cdf"
server_key_version: "encryptionkey@0"
I20260619 18:42:31.265074 23877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:31.277043 23877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:31.280001 23877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:31.280221 23877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:31.281654 23877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:31.281740 23877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:31.281807 23877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:31.281867 23877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:42:31 dist-test-slave-mld5 krb5kdc[23760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894551, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:31.293164 23877 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.65:45547
I20260619 18:42:31.293465 24007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.65:45547 every 8 connection(s)
I20260619 18:42:31.294296 23877 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
I20260619 18:42:31.296216 23854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:31.282495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:49385 (local address 127.21.143.126:45255)
0619 18:42:31.282658 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:31.282663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:31.283605 (+ 942us) server_negotiation.cc:407] Connection header received
0619 18:42:31.285011 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:31.285016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:31.285083 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:31.285199 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:31.286451 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.287013 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.288388 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.288715 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.288858 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:31.292202 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:31.292205 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:31.292207 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:31.292232 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:31.294028 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:31.294644 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:31.294648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:31.294650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:31.294754 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:31.295161 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:31.295164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:31.295165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:31.295424 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:31.295525 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:31.295990 (+ 465us) server_negotiation.cc:299] Negotiation successful
0619 18:42:31.296097 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 18:42:31.296988 23901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:31.282883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:45255 (local address 127.21.143.65:49385)
0619 18:42:31.283447 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:31.283506 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:31.284681 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:31.285384 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:31.285391 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:31.285738 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:31.286290 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:31.286312 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.287144 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.287148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:31.288255 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:31.288266 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.289039 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.289042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:31.289178 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:31.289989 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:31.290007 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:31.292017 (+ 2010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:31.294159 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:31.294164 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:31.294175 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:31.294522 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:31.294845 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:31.294850 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:31.294852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:31.295041 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:31.295528 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:31.295533 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:31.295773 (+ 240us) client_negotiation.cc:769] Sending connection context
0619 18:42:31.295922 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 18:42:31.296154 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":135,"threads_started":1}
I20260619 18:42:31.298143 24008 heartbeater.cc:347] Connected to a master server at 127.21.143.126:45255
I20260619 18:42:31.298374 24008 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:31.298936 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 23877
I20260619 18:42:31.298950 24008 heartbeater.cc:511] Master 127.21.143.126:45255 requested a full tablet report, sending...
I20260619 18:42:31.299073 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance
I20260619 18:42:31.299459 22077 external_mini_cluster.cc:1428] Setting key a67bcd2655bb5364d2996aa3cff86df7
I20260619 18:42:31.300611 23804 ts_manager.cc:194] Registered new tserver with Master: 217ce9884be4472583704a5d3c829f4e (127.21.143.65:45547)
I20260619 18:42:31.302260 23804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.65@KRBTEST.COM'} at 127.21.143.65:49385
WARNING: no policy specified for kudu/127.21.143.66@KRBTEST.COM; defaulting to no policy
I20260619 18:42:31.310077 23854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:31.300756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:53121 (local address 127.21.143.126:45255)
0619 18:42:31.300870 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:31.300875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:31.303158 (+ 2283us) server_negotiation.cc:407] Connection header received
0619 18:42:31.303357 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:31.303360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:31.303409 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:31.303494 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:31.304070 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.304588 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.305417 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.305645 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.305754 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:31.306860 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:31.306864 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:31.306867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:31.306890 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:31.308438 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:31.308895 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:31.308898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:31.308899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:31.308974 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:31.309268 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:31.309272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:31.309274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:31.309447 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:31.309553 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:31.309813 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 18:42:31.309923 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 18:42:31.310089 24015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:31.302725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:45255 (local address 127.21.143.65:53121)
0619 18:42:31.303050 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:31.303068 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:31.303185 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:31.303532 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:31.303538 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:31.303793 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:31.303950 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:31.303956 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.304731 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.304735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:31.305295 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:31.305302 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.305757 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.305761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:31.305865 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:31.306397 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:31.306409 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:31.306718 (+ 309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:31.308548 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:31.308553 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:31.308556 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:31.308754 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:31.309087 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:31.309090 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:31.309092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:31.309177 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:31.309545 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:31.309548 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:31.309644 (+ 96us) client_negotiation.cc:769] Sending connection context
0619 18:42:31.309749 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 18:42:31.309903 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab.
Entry for principal kudu/127.21.143.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab.
I20260619 18:42:31.325803 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.66:0
--local_ip_for_outbound_sockets=127.21.143.66
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:45255
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:41013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:31.435149 24017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:31.435518 24017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:31.435603 24017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:31.439756 24017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:31.439852 24017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:31.439906 24017 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 18:42:31.439920 24017 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:42:31.439965 24017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.66
I20260619 18:42:31.445392 24017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:41013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.143.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.143.126:45255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.66
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:31.446390 24017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:31.447366 24017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:31.456554 24023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:31.456554 24025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:31.456581 24022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:31.456712 24017 server_base.cc:1034] running on GCE node
I20260619 18:42:31.457194 24017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:31.457777 24017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:31.458966 24017 hybrid_clock.cc:648] HybridClock initialized: now 1781894551458951 us; error 26 us; skew 500 ppm
Jun 19 18:42:31 dist-test-slave-mld5 krb5kdc[23760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894551, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:31.461687 24017 init.cc:377] Logged in from keytab as kudu/127.21.143.66@KRBTEST.COM (short username kudu)
I20260619 18:42:31.462494 24017 webserver.cc:460] Webserver started at http://127.21.143.66:45063/ using document root <none> and password file <none>
I20260619 18:42:31.463047 24017 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:31.463135 24017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:31.463419 24017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:31.465381 24017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-1/data/instance:
uuid: "1bd7e55a8bfd4853985e44f1fb798167"
format_stamp: "Formatted at 2026-06-19 18:42:31 on dist-test-slave-mld5"
server_key: "54930825edf3c9ed4610e480c2c525cf"
server_key_iv: "313c8bbc0420b2e41a4ceac0d1b27a6c"
server_key_version: "encryptionkey@0"
I20260619 18:42:31.465924 24017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance:
uuid: "1bd7e55a8bfd4853985e44f1fb798167"
format_stamp: "Formatted at 2026-06-19 18:42:31 on dist-test-slave-mld5"
server_key: "54930825edf3c9ed4610e480c2c525cf"
server_key_iv: "313c8bbc0420b2e41a4ceac0d1b27a6c"
server_key_version: "encryptionkey@0"
I20260619 18:42:31.469508 24017 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260619 18:42:31.471820 24032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:31.472970 24017 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:42:31.473096 24017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-1/wal
uuid: "1bd7e55a8bfd4853985e44f1fb798167"
format_stamp: "Formatted at 2026-06-19 18:42:31 on dist-test-slave-mld5"
server_key: "54930825edf3c9ed4610e480c2c525cf"
server_key_iv: "313c8bbc0420b2e41a4ceac0d1b27a6c"
server_key_version: "encryptionkey@0"
I20260619 18:42:31.473201 24017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:31.488070 24017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:31.491127 24017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:31.491355 24017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:31.492765 24017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:31.492846 24017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:31.492970 24017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:31.493022 24017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:42:31 dist-test-slave-mld5 krb5kdc[23760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894551, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:31.504369 24017 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.66:33331
I20260619 18:42:31.504398 24147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.66:33331 every 8 connection(s)
I20260619 18:42:31.505347 24017 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
I20260619 18:42:31.507264 23854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:31.493802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:45373 (local address 127.21.143.126:45255)
0619 18:42:31.493931 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:31.493934 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:31.494880 (+ 946us) server_negotiation.cc:407] Connection header received
0619 18:42:31.495957 (+ 1077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:31.495961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:31.496026 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:31.496107 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:31.497894 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.498777 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.500338 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.500636 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.500756 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:31.503521 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:31.503524 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:31.503527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:31.503559 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:31.505226 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:31.505808 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:31.505811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:31.505813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:31.505902 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:31.506287 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:31.506291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:31.506293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:31.506511 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:31.506615 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:31.507046 (+ 431us) server_negotiation.cc:299] Negotiation successful
0619 18:42:31.507147 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 18:42:31.508052 24041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:31.494253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:45255 (local address 127.21.143.66:45373)
0619 18:42:31.494734 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:31.494775 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:31.495684 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:31.496594 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:31.496604 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:31.497060 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:31.497754 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:31.497778 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.498959 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.498965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:31.500207 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:31.500216 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.500799 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.500803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:31.500998 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:31.501600 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:31.501614 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:31.503189 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:31.505339 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:31.505344 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:31.505358 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:31.505684 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:31.506010 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:31.506013 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:31.506015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:31.506180 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:31.506618 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:31.506623 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:31.506836 (+ 213us) client_negotiation.cc:769] Sending connection context
0619 18:42:31.506980 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 18:42:31.507208 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":131,"threads_started":1}
I20260619 18:42:31.509326 24148 heartbeater.cc:347] Connected to a master server at 127.21.143.126:45255
I20260619 18:42:31.509488 24148 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:31.510057 24148 heartbeater.cc:511] Master 127.21.143.126:45255 requested a full tablet report, sending...
I20260619 18:42:31.511063 23804 ts_manager.cc:194] Registered new tserver with Master: 1bd7e55a8bfd4853985e44f1fb798167 (127.21.143.66:33331)
I20260619 18:42:31.511593 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 24017
I20260619 18:42:31.511737 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance
I20260619 18:42:31.511895 23804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.66@KRBTEST.COM'} at 127.21.143.66:45373
I20260619 18:42:31.512096 22077 external_mini_cluster.cc:1428] Setting key 7eb9220fc7d9e3c76c3aceaae8ef0fe5
I20260619 18:42:31.519039 24154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:31.511444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:45255 (local address 127.21.143.66:58337)
0619 18:42:31.512294 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:31.512306 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:31.512412 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:31.512837 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:31.512840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:31.513032 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:31.513124 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:31.513129 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.513846 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.513849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:31.514412 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:31.514419 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.514857 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.514861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:31.514951 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:31.515496 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:31.515516 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:31.515823 (+ 307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:31.517540 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:31.517543 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:31.517546 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:31.517735 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:31.518020 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:31.518023 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:31.518024 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:31.518104 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:31.518634 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:31.518636 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:31.518705 (+ 69us) client_negotiation.cc:769] Sending connection context
0619 18:42:31.518781 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 18:42:31.518884 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"thread_start_us":87,"threads_started":1}
I20260619 18:42:31.519244 23854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:31.511496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:58337 (local address 127.21.143.126:45255)
0619 18:42:31.511616 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:31.511621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:31.512373 (+ 752us) server_negotiation.cc:407] Connection header received
0619 18:42:31.512668 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:31.512671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:31.512715 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:31.512806 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:31.513235 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.513718 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.514528 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.514743 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.514856 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:31.515957 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:31.515961 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:31.515964 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:31.515993 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:31.517321 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:31.517840 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:31.517845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:31.517847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:31.517936 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:31.518213 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:31.518216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:31.518220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:31.518529 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:31.518650 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:31.518874 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 18:42:31.519024 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
WARNING: no policy specified for kudu/127.21.143.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab.
Entry for principal kudu/127.21.143.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab.
I20260619 18:42:31.538481 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.67:0
--local_ip_for_outbound_sockets=127.21.143.67
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:45255
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:41013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:31.647351 24157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:31.647645 24157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:31.647725 24157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:31.651691 24157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:31.651783 24157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:31.651856 24157 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 18:42:31.651901 24157 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:42:31.651971 24157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.67
I20260619 18:42:31.657173 24157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:41013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.143.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.143.126:45255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.67
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:31.658239 24157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:31.659257 24157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:31.668689 24163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:31.668869 24157 server_base.cc:1034] running on GCE node
W20260619 18:42:31.668668 24162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:31.668664 24165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:31.669395 24157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:31.670037 24157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:31.671227 24157 hybrid_clock.cc:648] HybridClock initialized: now 1781894551671198 us; error 42 us; skew 500 ppm
Jun 19 18:42:31 dist-test-slave-mld5 krb5kdc[23760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894551, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:31.673821 24157 init.cc:377] Logged in from keytab as kudu/127.21.143.67@KRBTEST.COM (short username kudu)
I20260619 18:42:31.674683 24157 webserver.cc:460] Webserver started at http://127.21.143.67:37321/ using document root <none> and password file <none>
I20260619 18:42:31.675232 24157 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:31.675350 24157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:31.675580 24157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:31.677631 24157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-2/data/instance:
uuid: "0034a6438c7c475abf22bfc0f92af818"
format_stamp: "Formatted at 2026-06-19 18:42:31 on dist-test-slave-mld5"
server_key: "91a0a71b9659c029f355f25c00d95b21"
server_key_iv: "1bd6abc54fcf43e7d549dc55f7773d5b"
server_key_version: "encryptionkey@0"
I20260619 18:42:31.678174 24157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance:
uuid: "0034a6438c7c475abf22bfc0f92af818"
format_stamp: "Formatted at 2026-06-19 18:42:31 on dist-test-slave-mld5"
server_key: "91a0a71b9659c029f355f25c00d95b21"
server_key_iv: "1bd6abc54fcf43e7d549dc55f7773d5b"
server_key_version: "encryptionkey@0"
I20260619 18:42:31.681734 24157 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:42:31.684119 24172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:31.685278 24157 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:42:31.685420 24157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-2/wal
uuid: "0034a6438c7c475abf22bfc0f92af818"
format_stamp: "Formatted at 2026-06-19 18:42:31 on dist-test-slave-mld5"
server_key: "91a0a71b9659c029f355f25c00d95b21"
server_key_iv: "1bd6abc54fcf43e7d549dc55f7773d5b"
server_key_version: "encryptionkey@0"
I20260619 18:42:31.685530 24157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:31.702828 24157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:31.706487 24157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:31.706710 24157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:31.708400 24157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:31.708487 24157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:31.708575 24157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:31.708629 24157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:42:31 dist-test-slave-mld5 krb5kdc[23760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894551, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:31.720013 24157 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.67:46073
I20260619 18:42:31.720093 24287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.67:46073 every 8 connection(s)
I20260619 18:42:31.721144 24157 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
I20260619 18:42:31.723001 24181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:31.709919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:45255 (local address 127.21.143.67:34357)
0619 18:42:31.710422 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:31.710472 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:31.711525 (+ 1053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:31.712337 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:31.712343 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:31.712696 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:31.713162 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:31.713174 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.713994 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.713997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:31.715022 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:31.715032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.715673 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.715677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:31.715829 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:31.716442 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:31.716466 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:31.718105 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:31.720162 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:31.720167 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:31.720181 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:31.720503 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:31.720778 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:31.720784 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:31.720789 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:31.720948 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:31.721359 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:31.721365 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:31.721646 (+ 281us) client_negotiation.cc:769] Sending connection context
0619 18:42:31.721842 (+ 196us) client_negotiation.cc:240] Negotiation successful
0619 18:42:31.722167 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":133,"threads_started":1}
I20260619 18:42:31.723629 23854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:31.709353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:34357 (local address 127.21.143.126:45255)
0619 18:42:31.709511 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:31.709517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:31.710584 (+ 1067us) server_negotiation.cc:407] Connection header received
0619 18:42:31.711787 (+ 1203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:31.711793 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:31.711882 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:31.711946 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:31.713308 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.713856 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.715147 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.715480 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.715602 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:31.718271 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:31.718274 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:31.718276 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:31.718299 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:31.720015 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:31.720601 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:31.720604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:31.720605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:31.720680 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:31.721037 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:31.721039 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:31.721040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:31.721244 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:31.721362 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:31.723423 (+ 2061us) server_negotiation.cc:299] Negotiation successful
0619 18:42:31.723523 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 18:42:31.724525 24288 heartbeater.cc:347] Connected to a master server at 127.21.143.126:45255
I20260619 18:42:31.724685 24288 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:31.724809 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 24157
I20260619 18:42:31.724929 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance
I20260619 18:42:31.725209 24288 heartbeater.cc:511] Master 127.21.143.126:45255 requested a full tablet report, sending...
I20260619 18:42:31.725265 22077 external_mini_cluster.cc:1428] Setting key bb8a8d31bc73ea03d97fd8762af3710b
I20260619 18:42:31.726209 23804 ts_manager.cc:194] Registered new tserver with Master: 0034a6438c7c475abf22bfc0f92af818 (127.21.143.67:46073)
I20260619 18:42:31.726895 23804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.67@KRBTEST.COM'} at 127.21.143.67:34357
I20260619 18:42:31.727620 22077 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:42:31.733940 23854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:31.727275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:60277 (local address 127.21.143.126:45255)
0619 18:42:31.727405 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:31.727410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:31.727422 (+ 12us) server_negotiation.cc:407] Connection header received
0619 18:42:31.727454 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:31.727457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:31.727496 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:31.727566 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:31.728125 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.728622 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.729473 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.729707 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.729813 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:31.730850 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:31.730853 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:31.730856 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:31.730879 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:31.732365 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:31.732780 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:31.732783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:31.732785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:31.732868 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:31.733152 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:31.733155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:31.733157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:31.733330 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:31.733436 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:31.733686 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 18:42:31.733793 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 18:42:31.733952 24294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:31.726859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:45255 (local address 127.21.143.67:60277)
0619 18:42:31.727118 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:31.727134 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:31.727219 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:31.727665 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:31.727667 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:31.727893 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:31.728012 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:31.728018 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.728740 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.728743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:31.729360 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:31.729369 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.729815 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.729818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:31.729915 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:31.730411 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:31.730424 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:31.730717 (+ 293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:31.732473 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:31.732480 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:31.732482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:31.732673 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:31.732970 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:31.732974 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:31.732976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:31.733055 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:31.733438 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:31.733443 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:31.733529 (+ 86us) client_negotiation.cc:769] Sending connection context
0619 18:42:31.733632 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 18:42:31.733771 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":1,"thread_start_us":141,"threads_started":1}
Jun 19 18:42:31 dist-test-slave-mld5 krb5kdc[23760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 18:42:31 dist-test-slave-mld5 krb5kdc[23760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:31.753665 23854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:31.745813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59086 (local address 127.21.143.126:45255)
0619 18:42:31.745953 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:31.745956 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:31.745967 (+ 11us) server_negotiation.cc:407] Connection header received
0619 18:42:31.746005 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:31.746007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:31.746044 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:31.746130 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:31.746810 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.747417 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.748282 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.748503 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.748611 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:31.750956 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:31.750959 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:31.750961 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:31.750986 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:31.752241 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:31.752604 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:31.752607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:31.752608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:31.752677 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:31.752898 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:31.752900 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:31.752901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:31.753059 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:31.753152 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:31.753346 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 18:42:31.753453 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 18:42:31.756763 23805 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:42:31.759850 23805 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:42:31.774080 24306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:31.768299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53296 (local address 127.21.143.66:33331)
0619 18:42:31.768713 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:31.768725 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:31.768750 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:42:31.768948 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:31.768953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:31.769196 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:31.769352 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:31.769837 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.770949 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.772515 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.773362 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.773505 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:31.773689 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:31.773823 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 18:42:31.773920 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":136,"threads_started":1}
I20260619 18:42:31.774068 24305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:31.768178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:33331 (local address 127.0.0.1:53296)
0619 18:42:31.768567 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:31.768620 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:31.768747 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:31.769533 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:31.769540 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:31.769568 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:31.769715 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:31.769722 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.771046 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.771052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:31.772387 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:31.772396 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.773522 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.773525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:31.773634 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:31.773676 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 18:42:31.773790 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 18:42:31.773915 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":70,"threads_started":1}
I20260619 18:42:31.774675 24307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:31.768928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:45547 (local address 127.0.0.1:36616)
0619 18:42:31.769251 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:31.769264 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:31.769350 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:31.770513 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:31.770518 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:31.770541 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:31.770657 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:31.770664 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.772097 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.772100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:31.773049 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:31.773058 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.774265 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.774272 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:31.774376 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:31.774398 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 18:42:31.774462 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 18:42:31.774541 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":40,"thread_start_us":110,"threads_started":1}
I20260619 18:42:31.775243 24309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:31.769805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36616 (local address 127.21.143.65:45547)
0619 18:42:31.770077 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:31.770095 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:31.770120 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:42:31.770191 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:31.770196 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:31.770374 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:31.770511 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:31.770764 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.771998 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.773165 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.773942 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.774045 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:31.774200 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:31.774981 (+ 781us) server_negotiation.cc:299] Negotiation successful
0619 18:42:31.775092 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":81,"threads_started":1}
I20260619 18:42:31.774844 24308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:31.769267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:46073 (local address 127.0.0.1:51634)
0619 18:42:31.769533 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:31.769545 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:31.769625 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:31.770650 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:31.770653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:31.770670 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:31.770766 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:31.770771 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.772152 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.772155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:31.773327 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:31.773338 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.774451 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.774455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:31.774561 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:31.774589 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 18:42:31.774650 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 18:42:31.774717 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":2,"spinlock_wait_cycles":1280,"thread_start_us":84,"threads_started":1}
I20260619 18:42:31.775636 24310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:31.769801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51634 (local address 127.21.143.67:46073)
0619 18:42:31.770176 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:31.770185 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:31.770203 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:42:31.770263 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:31.770266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:31.770438 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:31.770603 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:31.770877 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.772032 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.773462 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.774240 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.774354 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:31.775234 (+ 880us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:31.775333 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 18:42:31.775467 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":106,"threads_started":1}
I20260619 18:42:31.777590 24082 tablet_service.cc:1467] Processing CreateTablet for tablet 4090fee628da4847827ad23e013e1ef5 (DEFAULT_TABLE table=test-table [id=d011ff9f85114bd7bfb287e3838d2a92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:42:31.778035 23942 tablet_service.cc:1467] Processing CreateTablet for tablet 4090fee628da4847827ad23e013e1ef5 (DEFAULT_TABLE table=test-table [id=d011ff9f85114bd7bfb287e3838d2a92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:42:31.778756 24082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4090fee628da4847827ad23e013e1ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:31.778530 24222 tablet_service.cc:1467] Processing CreateTablet for tablet 4090fee628da4847827ad23e013e1ef5 (DEFAULT_TABLE table=test-table [id=d011ff9f85114bd7bfb287e3838d2a92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:42:31.779120 23942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4090fee628da4847827ad23e013e1ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:31.779517 24222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4090fee628da4847827ad23e013e1ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:31.785470 24311 tablet_bootstrap.cc:492] T 4090fee628da4847827ad23e013e1ef5 P 217ce9884be4472583704a5d3c829f4e: Bootstrap starting.
I20260619 18:42:31.786756 24313 tablet_bootstrap.cc:492] T 4090fee628da4847827ad23e013e1ef5 P 0034a6438c7c475abf22bfc0f92af818: Bootstrap starting.
I20260619 18:42:31.787551 24312 tablet_bootstrap.cc:492] T 4090fee628da4847827ad23e013e1ef5 P 1bd7e55a8bfd4853985e44f1fb798167: Bootstrap starting.
I20260619 18:42:31.787868 24311 tablet_bootstrap.cc:654] T 4090fee628da4847827ad23e013e1ef5 P 217ce9884be4472583704a5d3c829f4e: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:31.788978 24313 tablet_bootstrap.cc:654] T 4090fee628da4847827ad23e013e1ef5 P 0034a6438c7c475abf22bfc0f92af818: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:31.788745 24311 log.cc:824] T 4090fee628da4847827ad23e013e1ef5 P 217ce9884be4472583704a5d3c829f4e: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:31.789569 24312 tablet_bootstrap.cc:654] T 4090fee628da4847827ad23e013e1ef5 P 1bd7e55a8bfd4853985e44f1fb798167: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:31.789914 24313 log.cc:824] T 4090fee628da4847827ad23e013e1ef5 P 0034a6438c7c475abf22bfc0f92af818: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:31.790508 24312 log.cc:824] T 4090fee628da4847827ad23e013e1ef5 P 1bd7e55a8bfd4853985e44f1fb798167: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:31.791646 24311 tablet_bootstrap.cc:492] T 4090fee628da4847827ad23e013e1ef5 P 217ce9884be4472583704a5d3c829f4e: No bootstrap required, opened a new log
I20260619 18:42:31.791857 24311 ts_tablet_manager.cc:1397] T 4090fee628da4847827ad23e013e1ef5 P 217ce9884be4472583704a5d3c829f4e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 18:42:31.791927 24313 tablet_bootstrap.cc:492] T 4090fee628da4847827ad23e013e1ef5 P 0034a6438c7c475abf22bfc0f92af818: No bootstrap required, opened a new log
I20260619 18:42:31.792135 24313 ts_tablet_manager.cc:1397] T 4090fee628da4847827ad23e013e1ef5 P 0034a6438c7c475abf22bfc0f92af818: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 18:42:31.792379 24312 tablet_bootstrap.cc:492] T 4090fee628da4847827ad23e013e1ef5 P 1bd7e55a8bfd4853985e44f1fb798167: No bootstrap required, opened a new log
I20260619 18:42:31.792570 24312 ts_tablet_manager.cc:1397] T 4090fee628da4847827ad23e013e1ef5 P 1bd7e55a8bfd4853985e44f1fb798167: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 18:42:31.795364 24312 raft_consensus.cc:348] T 4090fee628da4847827ad23e013e1ef5 P 1bd7e55a8bfd4853985e44f1fb798167 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:31.795655 24312 raft_consensus.cc:374] T 4090fee628da4847827ad23e013e1ef5 P 1bd7e55a8bfd4853985e44f1fb798167 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:31.795707 24312 raft_consensus.cc:729] T 4090fee628da4847827ad23e013e1ef5 P 1bd7e55a8bfd4853985e44f1fb798167 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bd7e55a8bfd4853985e44f1fb798167, State: Initialized, Role: FOLLOWER
I20260619 18:42:31.795900 24313 raft_consensus.cc:348] T 4090fee628da4847827ad23e013e1ef5 P 0034a6438c7c475abf22bfc0f92af818 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:31.795957 24311 raft_consensus.cc:348] T 4090fee628da4847827ad23e013e1ef5 P 217ce9884be4472583704a5d3c829f4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:31.796160 24313 raft_consensus.cc:374] T 4090fee628da4847827ad23e013e1ef5 P 0034a6438c7c475abf22bfc0f92af818 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:31.796206 24313 raft_consensus.cc:729] T 4090fee628da4847827ad23e013e1ef5 P 0034a6438c7c475abf22bfc0f92af818 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0034a6438c7c475abf22bfc0f92af818, State: Initialized, Role: FOLLOWER
I20260619 18:42:31.796169 24312 consensus_queue.cc:260] T 4090fee628da4847827ad23e013e1ef5 P 1bd7e55a8bfd4853985e44f1fb798167 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:31.796437 24311 raft_consensus.cc:374] T 4090fee628da4847827ad23e013e1ef5 P 217ce9884be4472583704a5d3c829f4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:31.796550 24311 raft_consensus.cc:729] T 4090fee628da4847827ad23e013e1ef5 P 217ce9884be4472583704a5d3c829f4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 217ce9884be4472583704a5d3c829f4e, State: Initialized, Role: FOLLOWER
I20260619 18:42:31.796696 24313 consensus_queue.cc:260] T 4090fee628da4847827ad23e013e1ef5 P 0034a6438c7c475abf22bfc0f92af818 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:31.796988 24311 consensus_queue.cc:260] T 4090fee628da4847827ad23e013e1ef5 P 217ce9884be4472583704a5d3c829f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:31.797256 24312 ts_tablet_manager.cc:1428] T 4090fee628da4847827ad23e013e1ef5 P 1bd7e55a8bfd4853985e44f1fb798167: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260619 18:42:31.797432 24288 heartbeater.cc:503] Master 127.21.143.126:45255 was elected leader, sending a full tablet report...
I20260619 18:42:31.797714 24313 ts_tablet_manager.cc:1428] T 4090fee628da4847827ad23e013e1ef5 P 0034a6438c7c475abf22bfc0f92af818: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 18:42:31.797842 24148 heartbeater.cc:503] Master 127.21.143.126:45255 was elected leader, sending a full tablet report...
I20260619 18:42:31.797914 24311 ts_tablet_manager.cc:1428] T 4090fee628da4847827ad23e013e1ef5 P 217ce9884be4472583704a5d3c829f4e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 18:42:31.798266 24008 heartbeater.cc:503] Master 127.21.143.126:45255 was elected leader, sending a full tablet report...
W20260619 18:42:31.972148 24289 tablet.cc:2369] T 4090fee628da4847827ad23e013e1ef5 P 0034a6438c7c475abf22bfc0f92af818: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:42:31.986043 24317 raft_consensus.cc:482] T 4090fee628da4847827ad23e013e1ef5 P 1bd7e55a8bfd4853985e44f1fb798167 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:42:31.986294 24317 raft_consensus.cc:504] T 4090fee628da4847827ad23e013e1ef5 P 1bd7e55a8bfd4853985e44f1fb798167 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:31.987864 24317 leader_election.cc:288] T 4090fee628da4847827ad23e013e1ef5 P 1bd7e55a8bfd4853985e44f1fb798167 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 217ce9884be4472583704a5d3c829f4e (127.21.143.65:45547), 0034a6438c7c475abf22bfc0f92af818 (127.21.143.67:46073)
I20260619 18:42:31.995410 24041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:31.988312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:45547 (local address 127.21.143.66:57683)
0619 18:42:31.988435 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:31.988457 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:31.988564 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:31.991164 (+ 2600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:31.991167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:31.991185 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:31.991280 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:31.991285 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.992336 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.992340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:31.993849 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:31.993866 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.994862 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.994867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:31.995015 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:31.995044 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 18:42:31.995121 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 18:42:31.995193 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":69}
I20260619 18:42:31.995540 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:31.990697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:46073 (local address 127.21.143.66:45519)
0619 18:42:31.991249 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:31.991263 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:31.991435 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:31.991704 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:31.991709 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:31.991731 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:31.991872 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:31.991882 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.992895 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.992900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:31.993838 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:31.993847 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.994673 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.994680 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:31.995171 (+ 491us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:31.995189 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 18:42:31.995287 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 18:42:31.995373 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":20,"thread_start_us":80,"threads_started":1}
I20260619 18:42:31.995635 24310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:31.990767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:45519 (local address 127.21.143.67:46073)
0619 18:42:31.991113 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:31.991117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:31.991360 (+ 243us) server_negotiation.cc:407] Connection header received
0619 18:42:31.991554 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:31.991557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:31.991603 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:31.991677 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:31.991997 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.992781 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.993952 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.994548 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.994664 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:31.994715 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:31.995392 (+ 677us) server_negotiation.cc:299] Negotiation successful
0619 18:42:31.995484 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263}
I20260619 18:42:31.996165 24309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:31.990542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:57683 (local address 127.21.143.65:45547)
0619 18:42:31.990687 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:31.990692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:31.990711 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:42:31.990775 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:31.990781 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:31.990876 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:31.990969 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:31.991429 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.992144 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.994011 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:31.994624 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:31.994744 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:31.994794 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:31.996016 (+ 1222us) server_negotiation.cc:299] Negotiation successful
0619 18:42:31.996079 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 18:42:31.996713 23962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4090fee628da4847827ad23e013e1ef5" candidate_uuid: "1bd7e55a8bfd4853985e44f1fb798167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "217ce9884be4472583704a5d3c829f4e" is_pre_election: true
I20260619 18:42:31.996718 24242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4090fee628da4847827ad23e013e1ef5" candidate_uuid: "1bd7e55a8bfd4853985e44f1fb798167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0034a6438c7c475abf22bfc0f92af818" is_pre_election: true
I20260619 18:42:31.997040 23962 raft_consensus.cc:2454] T 4090fee628da4847827ad23e013e1ef5 P 217ce9884be4472583704a5d3c829f4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bd7e55a8bfd4853985e44f1fb798167 in term 0.
I20260619 18:42:31.997066 24242 raft_consensus.cc:2454] T 4090fee628da4847827ad23e013e1ef5 P 0034a6438c7c475abf22bfc0f92af818 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bd7e55a8bfd4853985e44f1fb798167 in term 0.
I20260619 18:42:31.997520 24033 leader_election.cc:302] T 4090fee628da4847827ad23e013e1ef5 P 1bd7e55a8bfd4853985e44f1fb798167 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bd7e55a8bfd4853985e44f1fb798167, 217ce9884be4472583704a5d3c829f4e; no voters:
I20260619 18:42:31.997820 24317 raft_consensus.cc:2781] T 4090fee628da4847827ad23e013e1ef5 P 1bd7e55a8bfd4853985e44f1fb798167 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:42:31.997924 24317 raft_consensus.cc:482] T 4090fee628da4847827ad23e013e1ef5 P 1bd7e55a8bfd4853985e44f1fb798167 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:42:31.997988 24317 raft_consensus.cc:3037] T 4090fee628da4847827ad23e013e1ef5 P 1bd7e55a8bfd4853985e44f1fb798167 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:31.999547 24317 raft_consensus.cc:504] T 4090fee628da4847827ad23e013e1ef5 P 1bd7e55a8bfd4853985e44f1fb798167 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:31.999993 24317 leader_election.cc:288] T 4090fee628da4847827ad23e013e1ef5 P 1bd7e55a8bfd4853985e44f1fb798167 [CANDIDATE]: Term 1 election: Requested vote from peers 217ce9884be4472583704a5d3c829f4e (127.21.143.65:45547), 0034a6438c7c475abf22bfc0f92af818 (127.21.143.67:46073)
I20260619 18:42:32.000396 23962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4090fee628da4847827ad23e013e1ef5" candidate_uuid: "1bd7e55a8bfd4853985e44f1fb798167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "217ce9884be4472583704a5d3c829f4e"
I20260619 18:42:32.000398 24242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4090fee628da4847827ad23e013e1ef5" candidate_uuid: "1bd7e55a8bfd4853985e44f1fb798167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0034a6438c7c475abf22bfc0f92af818"
I20260619 18:42:32.000504 23962 raft_consensus.cc:3037] T 4090fee628da4847827ad23e013e1ef5 P 217ce9884be4472583704a5d3c829f4e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:32.000542 24242 raft_consensus.cc:3037] T 4090fee628da4847827ad23e013e1ef5 P 0034a6438c7c475abf22bfc0f92af818 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:32.001552 23962 raft_consensus.cc:2454] T 4090fee628da4847827ad23e013e1ef5 P 217ce9884be4472583704a5d3c829f4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bd7e55a8bfd4853985e44f1fb798167 in term 1.
I20260619 18:42:32.001899 24033 leader_election.cc:302] T 4090fee628da4847827ad23e013e1ef5 P 1bd7e55a8bfd4853985e44f1fb798167 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bd7e55a8bfd4853985e44f1fb798167, 217ce9884be4472583704a5d3c829f4e; no voters:
I20260619 18:42:32.002044 24242 raft_consensus.cc:2454] T 4090fee628da4847827ad23e013e1ef5 P 0034a6438c7c475abf22bfc0f92af818 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bd7e55a8bfd4853985e44f1fb798167 in term 1.
I20260619 18:42:32.002080 24317 raft_consensus.cc:2781] T 4090fee628da4847827ad23e013e1ef5 P 1bd7e55a8bfd4853985e44f1fb798167 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:32.002316 24317 raft_consensus.cc:686] T 4090fee628da4847827ad23e013e1ef5 P 1bd7e55a8bfd4853985e44f1fb798167 [term 1 LEADER]: Becoming Leader. State: Replica: 1bd7e55a8bfd4853985e44f1fb798167, State: Running, Role: LEADER
I20260619 18:42:32.002672 24317 consensus_queue.cc:237] T 4090fee628da4847827ad23e013e1ef5 P 1bd7e55a8bfd4853985e44f1fb798167 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:32.006063 23804 catalog_manager.cc:5495] T 4090fee628da4847827ad23e013e1ef5 P 1bd7e55a8bfd4853985e44f1fb798167 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bd7e55a8bfd4853985e44f1fb798167 (127.21.143.66). New cstate: current_term: 1 leader_uuid: "1bd7e55a8bfd4853985e44f1fb798167" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } health_report { overall_health: UNKNOWN } } }
W20260619 18:42:32.006973 24149 tablet.cc:2369] T 4090fee628da4847827ad23e013e1ef5 P 1bd7e55a8bfd4853985e44f1fb798167: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:42:32.045679 24009 tablet.cc:2369] T 4090fee628da4847827ad23e013e1ef5 P 217ce9884be4472583704a5d3c829f4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 18:42:32 dist-test-slave-mld5 krb5kdc[23760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894550, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:32.093787 23854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:32.081032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59100 (local address 127.21.143.126:45255)
0619 18:42:32.081191 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:32.081196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:32.081380 (+ 184us) server_negotiation.cc:407] Connection header received
0619 18:42:32.081539 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:32.081543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:32.081589 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:32.081671 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:32.082354 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:32.082880 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:32.083733 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:32.083949 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:32.084059 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:32.090861 (+ 6802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:32.090864 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:32.090867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:32.090888 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:32.092228 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:32.092701 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:32.092704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:32.092705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:32.092773 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:32.093036 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:32.093038 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:32.093039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:32.093202 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:32.093273 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:32.093540 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 18:42:32.093658 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":56}
I20260619 18:42:32.093792 24328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:32.080994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:45255 (local address 127.0.0.1:59100)
0619 18:42:32.081286 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:32.081297 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:32.081443 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:32.081691 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:32.081694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:32.082147 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:32.082233 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:32.082239 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:32.082985 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:32.082992 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:32.083613 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:32.083622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:32.084058 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:32.084061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:32.084161 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:32.088991 (+ 4830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:32.089003 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:32.090719 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:32.092370 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:32.092377 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:32.092383 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:32.092613 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:32.092869 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:32.092871 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:32.092873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:32.092962 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:32.093286 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:32.093291 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:32.093397 (+ 106us) client_negotiation.cc:769] Sending connection context
0619 18:42:32.093485 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 18:42:32.093609 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":1,"thread_start_us":108,"threads_started":1}
I20260619 18:42:32.097923 23804 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:59100:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 18:42:32.098681 23804 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:42:32.103541 24082 tablet_service.cc:1467] Processing CreateTablet for tablet ed78b75fd64a472ba94c708de15f90b8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7ba06aee214a4f6dba93bd802545b055]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:42:32.103649 24222 tablet_service.cc:1467] Processing CreateTablet for tablet ed78b75fd64a472ba94c708de15f90b8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7ba06aee214a4f6dba93bd802545b055]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:42:32.103462 23942 tablet_service.cc:1467] Processing CreateTablet for tablet ed78b75fd64a472ba94c708de15f90b8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7ba06aee214a4f6dba93bd802545b055]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:42:32.104218 23942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed78b75fd64a472ba94c708de15f90b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:32.104218 24222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed78b75fd64a472ba94c708de15f90b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:32.104223 24082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed78b75fd64a472ba94c708de15f90b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:32.106564 24312 tablet_bootstrap.cc:492] T ed78b75fd64a472ba94c708de15f90b8 P 1bd7e55a8bfd4853985e44f1fb798167: Bootstrap starting.
I20260619 18:42:32.107069 24311 tablet_bootstrap.cc:492] T ed78b75fd64a472ba94c708de15f90b8 P 217ce9884be4472583704a5d3c829f4e: Bootstrap starting.
I20260619 18:42:32.107472 24312 tablet_bootstrap.cc:654] T ed78b75fd64a472ba94c708de15f90b8 P 1bd7e55a8bfd4853985e44f1fb798167: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:32.107947 24311 tablet_bootstrap.cc:654] T ed78b75fd64a472ba94c708de15f90b8 P 217ce9884be4472583704a5d3c829f4e: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:32.108460 24313 tablet_bootstrap.cc:492] T ed78b75fd64a472ba94c708de15f90b8 P 0034a6438c7c475abf22bfc0f92af818: Bootstrap starting.
I20260619 18:42:32.108667 24312 tablet_bootstrap.cc:492] T ed78b75fd64a472ba94c708de15f90b8 P 1bd7e55a8bfd4853985e44f1fb798167: No bootstrap required, opened a new log
I20260619 18:42:32.108768 24312 ts_tablet_manager.cc:1397] T ed78b75fd64a472ba94c708de15f90b8 P 1bd7e55a8bfd4853985e44f1fb798167: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 18:42:32.109263 24311 tablet_bootstrap.cc:492] T ed78b75fd64a472ba94c708de15f90b8 P 217ce9884be4472583704a5d3c829f4e: No bootstrap required, opened a new log
I20260619 18:42:32.109227 24312 raft_consensus.cc:348] T ed78b75fd64a472ba94c708de15f90b8 P 1bd7e55a8bfd4853985e44f1fb798167 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:32.109331 24312 raft_consensus.cc:374] T ed78b75fd64a472ba94c708de15f90b8 P 1bd7e55a8bfd4853985e44f1fb798167 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:32.109340 24311 ts_tablet_manager.cc:1397] T ed78b75fd64a472ba94c708de15f90b8 P 217ce9884be4472583704a5d3c829f4e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 18:42:32.109369 24312 raft_consensus.cc:729] T ed78b75fd64a472ba94c708de15f90b8 P 1bd7e55a8bfd4853985e44f1fb798167 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bd7e55a8bfd4853985e44f1fb798167, State: Initialized, Role: FOLLOWER
I20260619 18:42:32.109483 24313 tablet_bootstrap.cc:654] T ed78b75fd64a472ba94c708de15f90b8 P 0034a6438c7c475abf22bfc0f92af818: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:32.109511 24312 consensus_queue.cc:260] T ed78b75fd64a472ba94c708de15f90b8 P 1bd7e55a8bfd4853985e44f1fb798167 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:32.109764 24312 ts_tablet_manager.cc:1428] T ed78b75fd64a472ba94c708de15f90b8 P 1bd7e55a8bfd4853985e44f1fb798167: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 18:42:32.109791 24311 raft_consensus.cc:348] T ed78b75fd64a472ba94c708de15f90b8 P 217ce9884be4472583704a5d3c829f4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:32.109910 24311 raft_consensus.cc:374] T ed78b75fd64a472ba94c708de15f90b8 P 217ce9884be4472583704a5d3c829f4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:32.109962 24311 raft_consensus.cc:729] T ed78b75fd64a472ba94c708de15f90b8 P 217ce9884be4472583704a5d3c829f4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 217ce9884be4472583704a5d3c829f4e, State: Initialized, Role: FOLLOWER
I20260619 18:42:32.110074 24311 consensus_queue.cc:260] T ed78b75fd64a472ba94c708de15f90b8 P 217ce9884be4472583704a5d3c829f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:32.110332 24311 ts_tablet_manager.cc:1428] T ed78b75fd64a472ba94c708de15f90b8 P 217ce9884be4472583704a5d3c829f4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:42:32.110436 24319 tablet_replica.cc:439] T ed78b75fd64a472ba94c708de15f90b8 P 217ce9884be4472583704a5d3c829f4e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } } }
I20260619 18:42:32.110574 24319 tablet_replica.cc:442] T ed78b75fd64a472ba94c708de15f90b8 P 217ce9884be4472583704a5d3c829f4e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:42:32.110535 24317 tablet_replica.cc:439] T ed78b75fd64a472ba94c708de15f90b8 P 1bd7e55a8bfd4853985e44f1fb798167: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } } }
I20260619 18:42:32.110652 24317 tablet_replica.cc:442] T ed78b75fd64a472ba94c708de15f90b8 P 1bd7e55a8bfd4853985e44f1fb798167: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:42:32.111727 24313 tablet_bootstrap.cc:492] T ed78b75fd64a472ba94c708de15f90b8 P 0034a6438c7c475abf22bfc0f92af818: No bootstrap required, opened a new log
I20260619 18:42:32.111814 24313 ts_tablet_manager.cc:1397] T ed78b75fd64a472ba94c708de15f90b8 P 0034a6438c7c475abf22bfc0f92af818: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 18:42:32.112272 24313 raft_consensus.cc:348] T ed78b75fd64a472ba94c708de15f90b8 P 0034a6438c7c475abf22bfc0f92af818 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:32.112411 24313 raft_consensus.cc:374] T ed78b75fd64a472ba94c708de15f90b8 P 0034a6438c7c475abf22bfc0f92af818 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:32.112454 24313 raft_consensus.cc:729] T ed78b75fd64a472ba94c708de15f90b8 P 0034a6438c7c475abf22bfc0f92af818 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0034a6438c7c475abf22bfc0f92af818, State: Initialized, Role: FOLLOWER
I20260619 18:42:32.112573 24313 consensus_queue.cc:260] T ed78b75fd64a472ba94c708de15f90b8 P 0034a6438c7c475abf22bfc0f92af818 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:32.112879 24313 ts_tablet_manager.cc:1428] T ed78b75fd64a472ba94c708de15f90b8 P 0034a6438c7c475abf22bfc0f92af818: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 18:42:32.113363 24318 tablet_replica.cc:439] T ed78b75fd64a472ba94c708de15f90b8 P 0034a6438c7c475abf22bfc0f92af818: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } } }
I20260619 18:42:32.113502 24318 tablet_replica.cc:442] T ed78b75fd64a472ba94c708de15f90b8 P 0034a6438c7c475abf22bfc0f92af818: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:42:32.151501 24317 raft_consensus.cc:482] T ed78b75fd64a472ba94c708de15f90b8 P 1bd7e55a8bfd4853985e44f1fb798167 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:42:32.151624 24317 raft_consensus.cc:504] T ed78b75fd64a472ba94c708de15f90b8 P 1bd7e55a8bfd4853985e44f1fb798167 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:32.152051 24317 leader_election.cc:288] T ed78b75fd64a472ba94c708de15f90b8 P 1bd7e55a8bfd4853985e44f1fb798167 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 217ce9884be4472583704a5d3c829f4e (127.21.143.65:45547), 0034a6438c7c475abf22bfc0f92af818 (127.21.143.67:46073)
I20260619 18:42:32.152558 24242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed78b75fd64a472ba94c708de15f90b8" candidate_uuid: "1bd7e55a8bfd4853985e44f1fb798167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0034a6438c7c475abf22bfc0f92af818" is_pre_election: true
I20260619 18:42:32.152559 23962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed78b75fd64a472ba94c708de15f90b8" candidate_uuid: "1bd7e55a8bfd4853985e44f1fb798167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "217ce9884be4472583704a5d3c829f4e" is_pre_election: true
I20260619 18:42:32.152711 24242 raft_consensus.cc:2454] T ed78b75fd64a472ba94c708de15f90b8 P 0034a6438c7c475abf22bfc0f92af818 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bd7e55a8bfd4853985e44f1fb798167 in term 0.
I20260619 18:42:32.152711 23962 raft_consensus.cc:2454] T ed78b75fd64a472ba94c708de15f90b8 P 217ce9884be4472583704a5d3c829f4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bd7e55a8bfd4853985e44f1fb798167 in term 0.
I20260619 18:42:32.153005 24033 leader_election.cc:302] T ed78b75fd64a472ba94c708de15f90b8 P 1bd7e55a8bfd4853985e44f1fb798167 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0034a6438c7c475abf22bfc0f92af818, 1bd7e55a8bfd4853985e44f1fb798167; no voters:
I20260619 18:42:32.153182 24317 raft_consensus.cc:2781] T ed78b75fd64a472ba94c708de15f90b8 P 1bd7e55a8bfd4853985e44f1fb798167 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:42:32.153254 24317 raft_consensus.cc:482] T ed78b75fd64a472ba94c708de15f90b8 P 1bd7e55a8bfd4853985e44f1fb798167 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:42:32.153304 24317 raft_consensus.cc:3037] T ed78b75fd64a472ba94c708de15f90b8 P 1bd7e55a8bfd4853985e44f1fb798167 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:32.154359 24317 raft_consensus.cc:504] T ed78b75fd64a472ba94c708de15f90b8 P 1bd7e55a8bfd4853985e44f1fb798167 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:32.154676 24317 leader_election.cc:288] T ed78b75fd64a472ba94c708de15f90b8 P 1bd7e55a8bfd4853985e44f1fb798167 [CANDIDATE]: Term 1 election: Requested vote from peers 217ce9884be4472583704a5d3c829f4e (127.21.143.65:45547), 0034a6438c7c475abf22bfc0f92af818 (127.21.143.67:46073)
I20260619 18:42:32.155069 24242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed78b75fd64a472ba94c708de15f90b8" candidate_uuid: "1bd7e55a8bfd4853985e44f1fb798167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0034a6438c7c475abf22bfc0f92af818"
I20260619 18:42:32.155069 23962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed78b75fd64a472ba94c708de15f90b8" candidate_uuid: "1bd7e55a8bfd4853985e44f1fb798167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "217ce9884be4472583704a5d3c829f4e"
I20260619 18:42:32.155189 24242 raft_consensus.cc:3037] T ed78b75fd64a472ba94c708de15f90b8 P 0034a6438c7c475abf22bfc0f92af818 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:32.155189 23962 raft_consensus.cc:3037] T ed78b75fd64a472ba94c708de15f90b8 P 217ce9884be4472583704a5d3c829f4e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:32.156312 23962 raft_consensus.cc:2454] T ed78b75fd64a472ba94c708de15f90b8 P 217ce9884be4472583704a5d3c829f4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bd7e55a8bfd4853985e44f1fb798167 in term 1.
I20260619 18:42:32.156312 24242 raft_consensus.cc:2454] T ed78b75fd64a472ba94c708de15f90b8 P 0034a6438c7c475abf22bfc0f92af818 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bd7e55a8bfd4853985e44f1fb798167 in term 1.
I20260619 18:42:32.156601 24033 leader_election.cc:302] T ed78b75fd64a472ba94c708de15f90b8 P 1bd7e55a8bfd4853985e44f1fb798167 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bd7e55a8bfd4853985e44f1fb798167, 217ce9884be4472583704a5d3c829f4e; no voters:
I20260619 18:42:32.156795 24317 raft_consensus.cc:2781] T ed78b75fd64a472ba94c708de15f90b8 P 1bd7e55a8bfd4853985e44f1fb798167 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:32.156878 24317 raft_consensus.cc:686] T ed78b75fd64a472ba94c708de15f90b8 P 1bd7e55a8bfd4853985e44f1fb798167 [term 1 LEADER]: Becoming Leader. State: Replica: 1bd7e55a8bfd4853985e44f1fb798167, State: Running, Role: LEADER
I20260619 18:42:32.157033 24317 consensus_queue.cc:237] T ed78b75fd64a472ba94c708de15f90b8 P 1bd7e55a8bfd4853985e44f1fb798167 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:32.157646 24321 tablet_replica.cc:439] T ed78b75fd64a472ba94c708de15f90b8 P 1bd7e55a8bfd4853985e44f1fb798167: TxnStatusTablet state changed. Reason: New leader 1bd7e55a8bfd4853985e44f1fb798167. Latest consensus state: current_term: 1 leader_uuid: "1bd7e55a8bfd4853985e44f1fb798167" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } } }
I20260619 18:42:32.157757 24321 tablet_replica.cc:442] T ed78b75fd64a472ba94c708de15f90b8 P 1bd7e55a8bfd4853985e44f1fb798167: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:42:32.158058 24333 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 18:42:32.158311 24333 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 18:42:32.158610 23804 catalog_manager.cc:5495] T ed78b75fd64a472ba94c708de15f90b8 P 1bd7e55a8bfd4853985e44f1fb798167 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bd7e55a8bfd4853985e44f1fb798167 (127.21.143.66). New cstate: current_term: 1 leader_uuid: "1bd7e55a8bfd4853985e44f1fb798167" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } health_report { overall_health: UNKNOWN } } }
I20260619 18:42:32.177066 24328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:32.173050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:33331 (local address 127.0.0.1:53312)
0619 18:42:32.173178 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:32.173194 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:32.173283 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:32.173603 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:32.173604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:32.173611 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:42:32.173695 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:32.173699 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:32.174682 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:32.174685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:32.175260 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:32.175267 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:32.175878 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:32.175881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:32.175996 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:32.176036 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:42:32.176772 (+ 736us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:42:32.176782 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 18:42:32.176864 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 18:42:32.176938 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 18:42:32.177069 24306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:32.173160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53312 (local address 127.21.143.66:33331)
0619 18:42:32.173259 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:32.173263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:32.173276 (+ 13us) server_negotiation.cc:407] Connection header received
0619 18:42:32.173391 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:32.173394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:32.173489 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:32.173589 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:42:32.173793 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:32.174565 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:32.175423 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:32.175777 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:32.175878 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:32.176203 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:42:32.176689 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:42:32.176794 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:32.176879 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 18:42:32.176931 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 18:42:32.181802 23962 raft_consensus.cc:1261] T ed78b75fd64a472ba94c708de15f90b8 P 217ce9884be4472583704a5d3c829f4e [term 1 FOLLOWER]: Refusing update from remote peer 1bd7e55a8bfd4853985e44f1fb798167: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:42:32.181910 24242 raft_consensus.cc:1261] T ed78b75fd64a472ba94c708de15f90b8 P 0034a6438c7c475abf22bfc0f92af818 [term 1 FOLLOWER]: Refusing update from remote peer 1bd7e55a8bfd4853985e44f1fb798167: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:42:32.182502 24317 consensus_queue.cc:1035] T ed78b75fd64a472ba94c708de15f90b8 P 1bd7e55a8bfd4853985e44f1fb798167 [LEADER]: Connected to new peer: Peer: permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:42:32.182668 24321 consensus_queue.cc:1035] T ed78b75fd64a472ba94c708de15f90b8 P 1bd7e55a8bfd4853985e44f1fb798167 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:42:32.186092 24319 tablet_replica.cc:439] T ed78b75fd64a472ba94c708de15f90b8 P 217ce9884be4472583704a5d3c829f4e: TxnStatusTablet state changed. Reason: New leader 1bd7e55a8bfd4853985e44f1fb798167. Latest consensus state: current_term: 1 leader_uuid: "1bd7e55a8bfd4853985e44f1fb798167" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } } }
I20260619 18:42:32.186092 24318 tablet_replica.cc:439] T ed78b75fd64a472ba94c708de15f90b8 P 0034a6438c7c475abf22bfc0f92af818: TxnStatusTablet state changed. Reason: New leader 1bd7e55a8bfd4853985e44f1fb798167. Latest consensus state: current_term: 1 leader_uuid: "1bd7e55a8bfd4853985e44f1fb798167" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } } }
I20260619 18:42:32.186194 24319 tablet_replica.cc:442] T ed78b75fd64a472ba94c708de15f90b8 P 217ce9884be4472583704a5d3c829f4e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:42:32.186195 24318 tablet_replica.cc:442] T ed78b75fd64a472ba94c708de15f90b8 P 0034a6438c7c475abf22bfc0f92af818: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:42:32.190203 24321 tablet_replica.cc:439] T ed78b75fd64a472ba94c708de15f90b8 P 1bd7e55a8bfd4853985e44f1fb798167: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1bd7e55a8bfd4853985e44f1fb798167" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } } }
I20260619 18:42:32.190310 24321 tablet_replica.cc:442] T ed78b75fd64a472ba94c708de15f90b8 P 1bd7e55a8bfd4853985e44f1fb798167: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:42:32.190750 24318 tablet_replica.cc:439] T ed78b75fd64a472ba94c708de15f90b8 P 0034a6438c7c475abf22bfc0f92af818: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1bd7e55a8bfd4853985e44f1fb798167" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } } }
I20260619 18:42:32.190858 24318 tablet_replica.cc:442] T ed78b75fd64a472ba94c708de15f90b8 P 0034a6438c7c475abf22bfc0f92af818: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:42:32.191093 24337 mvcc.cc:205] Tried to move back new op lower bound from 7298640085730840576 to 7298640085636382720. Current Snapshot: MvccSnapshot[applied={T|T < 7298640085730840576}]
I20260619 18:42:32.191818 24336 mvcc.cc:205] Tried to move back new op lower bound from 7298640085730840576 to 7298640085636382720. Current Snapshot: MvccSnapshot[applied={T|T < 7298640085730840576}]
I20260619 18:42:32.192195 24317 tablet_replica.cc:439] T ed78b75fd64a472ba94c708de15f90b8 P 1bd7e55a8bfd4853985e44f1fb798167: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1bd7e55a8bfd4853985e44f1fb798167" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } } }
I20260619 18:42:32.192294 24317 tablet_replica.cc:442] T ed78b75fd64a472ba94c708de15f90b8 P 1bd7e55a8bfd4853985e44f1fb798167: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:42:32.194495 24319 tablet_replica.cc:439] T ed78b75fd64a472ba94c708de15f90b8 P 217ce9884be4472583704a5d3c829f4e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1bd7e55a8bfd4853985e44f1fb798167" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } } }
I20260619 18:42:32.194612 24319 tablet_replica.cc:442] T ed78b75fd64a472ba94c708de15f90b8 P 217ce9884be4472583704a5d3c829f4e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:42:32.201926 24306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:32.198497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53318 (local address 127.21.143.66:33331)
0619 18:42:32.198611 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:32.198615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:32.198686 (+ 71us) server_negotiation.cc:407] Connection header received
0619 18:42:32.198787 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:32.198789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:32.198833 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:32.198909 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:42:32.199241 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:32.200044 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:32.200873 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:32.201213 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:32.201311 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:32.201533 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:42:32.201611 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:42:32.201681 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:32.201749 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 18:42:32.201796 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 18:42:32.212119 24306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:32.208578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:38865 (local address 127.21.143.66:33331)
0619 18:42:32.208677 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:32.208680 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:32.208937 (+ 257us) server_negotiation.cc:407] Connection header received
0619 18:42:32.209047 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:32.209049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:32.209083 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:32.209139 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:42:32.209415 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:32.209911 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:32.211099 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:32.211282 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:32.211422 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:32.211732 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:42:32.211785 (+ 53us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:42:32.211846 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:32.211969 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 18:42:32.212003 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 18:42:32.212220 24348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:32.208512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:33331 (local address 127.21.143.66:38865)
0619 18:42:32.208856 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:32.208869 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:32.208972 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:32.209169 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:32.209171 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:32.209192 (+ 21us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:42:32.209314 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:32.209320 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:32.209996 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:32.210000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:32.210985 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:32.210995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:32.211433 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:32.211437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:32.211543 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:32.211590 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:42:32.211875 (+ 285us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:42:32.211880 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 18:42:32.211953 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 18:42:32.212030 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
I20260619 18:42:32.219094 23962 raft_consensus.cc:1261] T 4090fee628da4847827ad23e013e1ef5 P 217ce9884be4472583704a5d3c829f4e [term 1 FOLLOWER]: Refusing update from remote peer 1bd7e55a8bfd4853985e44f1fb798167: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:42:32.219208 24242 raft_consensus.cc:1261] T 4090fee628da4847827ad23e013e1ef5 P 0034a6438c7c475abf22bfc0f92af818 [term 1 FOLLOWER]: Refusing update from remote peer 1bd7e55a8bfd4853985e44f1fb798167: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:42:32.219520 24317 consensus_queue.cc:1035] T 4090fee628da4847827ad23e013e1ef5 P 1bd7e55a8bfd4853985e44f1fb798167 [LEADER]: Connected to new peer: Peer: permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:42:32.219660 24343 consensus_queue.cc:1035] T 4090fee628da4847827ad23e013e1ef5 P 1bd7e55a8bfd4853985e44f1fb798167 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:42:32.225688 24338 mvcc.cc:205] Tried to move back new op lower bound from 7298640085886164992 to 7298640085006069760. Current Snapshot: MvccSnapshot[applied={T|T < 7298640085886164992}]
I20260619 18:42:32.262724 23803 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:42:32.262879 23803 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:42:32.265168 23803 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 299c0445f59a43ad86d5be47c7d9a35c: Sending DeleteTablet for 3 replicas of tablet 4090fee628da4847827ad23e013e1ef5
I20260619 18:42:32.265913 24082 tablet_service.cc:1514] Processing DeleteTablet for tablet 4090fee628da4847827ad23e013e1ef5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:42:32 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:53296
I20260619 18:42:32.265926 23942 tablet_service.cc:1514] Processing DeleteTablet for tablet 4090fee628da4847827ad23e013e1ef5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:42:32 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:36616
I20260619 18:42:32.266044 24222 tablet_service.cc:1514] Processing DeleteTablet for tablet 4090fee628da4847827ad23e013e1ef5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:42:32 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:51634
I20260619 18:42:32.266747 24352 tablet_replica.cc:330] T 4090fee628da4847827ad23e013e1ef5 P 217ce9884be4472583704a5d3c829f4e: stopping tablet replica
I20260619 18:42:32.267057 24352 raft_consensus.cc:2229] T 4090fee628da4847827ad23e013e1ef5 P 217ce9884be4472583704a5d3c829f4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:42:32.266987 23803 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 18:42:32.267282 24352 raft_consensus.cc:2258] T 4090fee628da4847827ad23e013e1ef5 P 217ce9884be4472583704a5d3c829f4e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 18:42:32.267406 23803 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:42:32.268708 24352 ts_tablet_manager.cc:1905] T 4090fee628da4847827ad23e013e1ef5 P 217ce9884be4472583704a5d3c829f4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:42:32.271482 24354 tablet_replica.cc:330] T 4090fee628da4847827ad23e013e1ef5 P 0034a6438c7c475abf22bfc0f92af818: stopping tablet replica
I20260619 18:42:32.271494 24352 ts_tablet_manager.cc:1918] T 4090fee628da4847827ad23e013e1ef5 P 217ce9884be4472583704a5d3c829f4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:42:32.271579 24352 log.cc:1196] T 4090fee628da4847827ad23e013e1ef5 P 217ce9884be4472583704a5d3c829f4e: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-0/wal/wals/4090fee628da4847827ad23e013e1ef5
I20260619 18:42:32.271770 24353 tablet_replica.cc:330] T 4090fee628da4847827ad23e013e1ef5 P 1bd7e55a8bfd4853985e44f1fb798167: stopping tablet replica
I20260619 18:42:32.271917 24352 ts_tablet_manager.cc:1939] T 4090fee628da4847827ad23e013e1ef5 P 217ce9884be4472583704a5d3c829f4e: Deleting consensus metadata
I20260619 18:42:32.272251 24354 raft_consensus.cc:2229] T 4090fee628da4847827ad23e013e1ef5 P 0034a6438c7c475abf22bfc0f92af818 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:42:32.272313 23942 tablet_service.cc:1467] Processing CreateTablet for tablet 802130b8e94947758554432795f8f8e1 (DEFAULT_TABLE table=test-table [id=7ecad7dd7f3c4b5c88a0b548f26452aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:42:32.272557 23942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 802130b8e94947758554432795f8f8e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:32.272535 24353 raft_consensus.cc:2229] T 4090fee628da4847827ad23e013e1ef5 P 1bd7e55a8bfd4853985e44f1fb798167 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:42:32.273381 24354 raft_consensus.cc:2258] T 4090fee628da4847827ad23e013e1ef5 P 0034a6438c7c475abf22bfc0f92af818 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:42:32.273819 24353 raft_consensus.cc:2258] T 4090fee628da4847827ad23e013e1ef5 P 1bd7e55a8bfd4853985e44f1fb798167 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:42:32.274845 24311 tablet_bootstrap.cc:492] T 802130b8e94947758554432795f8f8e1 P 217ce9884be4472583704a5d3c829f4e: Bootstrap starting.
I20260619 18:42:32.275117 23789 catalog_manager.cc:4841] TS 217ce9884be4472583704a5d3c829f4e (127.21.143.65:45547): tablet 4090fee628da4847827ad23e013e1ef5 (table test-table [id=d011ff9f85114bd7bfb287e3838d2a92]) successfully deleted
I20260619 18:42:32.275705 24222 tablet_service.cc:1467] Processing CreateTablet for tablet 802130b8e94947758554432795f8f8e1 (DEFAULT_TABLE table=test-table [id=7ecad7dd7f3c4b5c88a0b548f26452aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:42:32.275821 24311 tablet_bootstrap.cc:654] T 802130b8e94947758554432795f8f8e1 P 217ce9884be4472583704a5d3c829f4e: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:32.275966 24222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 802130b8e94947758554432795f8f8e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:32.276058 24353 ts_tablet_manager.cc:1905] T 4090fee628da4847827ad23e013e1ef5 P 1bd7e55a8bfd4853985e44f1fb798167: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:42:32.277416 24354 ts_tablet_manager.cc:1905] T 4090fee628da4847827ad23e013e1ef5 P 0034a6438c7c475abf22bfc0f92af818: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:42:32.278344 24313 tablet_bootstrap.cc:492] T 802130b8e94947758554432795f8f8e1 P 0034a6438c7c475abf22bfc0f92af818: Bootstrap starting.
I20260619 18:42:32.279067 24353 ts_tablet_manager.cc:1918] T 4090fee628da4847827ad23e013e1ef5 P 1bd7e55a8bfd4853985e44f1fb798167: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:42:32.279212 24353 log.cc:1196] T 4090fee628da4847827ad23e013e1ef5 P 1bd7e55a8bfd4853985e44f1fb798167: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-1/wal/wals/4090fee628da4847827ad23e013e1ef5
I20260619 18:42:32.279289 24313 tablet_bootstrap.cc:654] T 802130b8e94947758554432795f8f8e1 P 0034a6438c7c475abf22bfc0f92af818: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:32.279664 24353 ts_tablet_manager.cc:1939] T 4090fee628da4847827ad23e013e1ef5 P 1bd7e55a8bfd4853985e44f1fb798167: Deleting consensus metadata
I20260619 18:42:32.280274 24082 tablet_service.cc:1467] Processing CreateTablet for tablet 802130b8e94947758554432795f8f8e1 (DEFAULT_TABLE table=test-table [id=7ecad7dd7f3c4b5c88a0b548f26452aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:42:32.280516 24082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 802130b8e94947758554432795f8f8e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:32.280869 23789 catalog_manager.cc:4841] TS 1bd7e55a8bfd4853985e44f1fb798167 (127.21.143.66:33331): tablet 4090fee628da4847827ad23e013e1ef5 (table test-table [id=d011ff9f85114bd7bfb287e3838d2a92]) successfully deleted
I20260619 18:42:32.280890 24354 ts_tablet_manager.cc:1918] T 4090fee628da4847827ad23e013e1ef5 P 0034a6438c7c475abf22bfc0f92af818: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:42:32.280974 24354 log.cc:1196] T 4090fee628da4847827ad23e013e1ef5 P 0034a6438c7c475abf22bfc0f92af818: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-2/wal/wals/4090fee628da4847827ad23e013e1ef5
I20260619 18:42:32.281267 24354 ts_tablet_manager.cc:1939] T 4090fee628da4847827ad23e013e1ef5 P 0034a6438c7c475abf22bfc0f92af818: Deleting consensus metadata
I20260619 18:42:32.282387 23789 catalog_manager.cc:4841] TS 0034a6438c7c475abf22bfc0f92af818 (127.21.143.67:46073): tablet 4090fee628da4847827ad23e013e1ef5 (table test-table [id=d011ff9f85114bd7bfb287e3838d2a92]) successfully deleted
I20260619 18:42:32.282603 24312 tablet_bootstrap.cc:492] T 802130b8e94947758554432795f8f8e1 P 1bd7e55a8bfd4853985e44f1fb798167: Bootstrap starting.
I20260619 18:42:32.282932 24311 tablet_bootstrap.cc:492] T 802130b8e94947758554432795f8f8e1 P 217ce9884be4472583704a5d3c829f4e: No bootstrap required, opened a new log
I20260619 18:42:32.283036 24311 ts_tablet_manager.cc:1397] T 802130b8e94947758554432795f8f8e1 P 217ce9884be4472583704a5d3c829f4e: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260619 18:42:32.283550 24312 tablet_bootstrap.cc:654] T 802130b8e94947758554432795f8f8e1 P 1bd7e55a8bfd4853985e44f1fb798167: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:32.283509 24311 raft_consensus.cc:348] T 802130b8e94947758554432795f8f8e1 P 217ce9884be4472583704a5d3c829f4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:32.283630 24311 raft_consensus.cc:374] T 802130b8e94947758554432795f8f8e1 P 217ce9884be4472583704a5d3c829f4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:32.283666 24311 raft_consensus.cc:729] T 802130b8e94947758554432795f8f8e1 P 217ce9884be4472583704a5d3c829f4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 217ce9884be4472583704a5d3c829f4e, State: Initialized, Role: FOLLOWER
I20260619 18:42:32.283778 24311 consensus_queue.cc:260] T 802130b8e94947758554432795f8f8e1 P 217ce9884be4472583704a5d3c829f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:32.284044 24311 ts_tablet_manager.cc:1428] T 802130b8e94947758554432795f8f8e1 P 217ce9884be4472583704a5d3c829f4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:42:32.284472 24313 tablet_bootstrap.cc:492] T 802130b8e94947758554432795f8f8e1 P 0034a6438c7c475abf22bfc0f92af818: No bootstrap required, opened a new log
I20260619 18:42:32.284576 24313 ts_tablet_manager.cc:1397] T 802130b8e94947758554432795f8f8e1 P 0034a6438c7c475abf22bfc0f92af818: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260619 18:42:32.284965 24313 raft_consensus.cc:348] T 802130b8e94947758554432795f8f8e1 P 0034a6438c7c475abf22bfc0f92af818 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:32.285075 24313 raft_consensus.cc:374] T 802130b8e94947758554432795f8f8e1 P 0034a6438c7c475abf22bfc0f92af818 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:32.285122 24313 raft_consensus.cc:729] T 802130b8e94947758554432795f8f8e1 P 0034a6438c7c475abf22bfc0f92af818 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0034a6438c7c475abf22bfc0f92af818, State: Initialized, Role: FOLLOWER
I20260619 18:42:32.285256 24313 consensus_queue.cc:260] T 802130b8e94947758554432795f8f8e1 P 0034a6438c7c475abf22bfc0f92af818 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:32.285513 24313 ts_tablet_manager.cc:1428] T 802130b8e94947758554432795f8f8e1 P 0034a6438c7c475abf22bfc0f92af818: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 18:42:32.288070 24312 tablet_bootstrap.cc:492] T 802130b8e94947758554432795f8f8e1 P 1bd7e55a8bfd4853985e44f1fb798167: No bootstrap required, opened a new log
I20260619 18:42:32.288154 24312 ts_tablet_manager.cc:1397] T 802130b8e94947758554432795f8f8e1 P 1bd7e55a8bfd4853985e44f1fb798167: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260619 18:42:32.288590 24312 raft_consensus.cc:348] T 802130b8e94947758554432795f8f8e1 P 1bd7e55a8bfd4853985e44f1fb798167 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:32.288722 24312 raft_consensus.cc:374] T 802130b8e94947758554432795f8f8e1 P 1bd7e55a8bfd4853985e44f1fb798167 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:32.288753 24312 raft_consensus.cc:729] T 802130b8e94947758554432795f8f8e1 P 1bd7e55a8bfd4853985e44f1fb798167 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bd7e55a8bfd4853985e44f1fb798167, State: Initialized, Role: FOLLOWER
I20260619 18:42:32.288872 24312 consensus_queue.cc:260] T 802130b8e94947758554432795f8f8e1 P 1bd7e55a8bfd4853985e44f1fb798167 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:32.289129 24312 ts_tablet_manager.cc:1428] T 802130b8e94947758554432795f8f8e1 P 1bd7e55a8bfd4853985e44f1fb798167: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:42:32.353889 24318 raft_consensus.cc:482] T 802130b8e94947758554432795f8f8e1 P 0034a6438c7c475abf22bfc0f92af818 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:42:32.354036 24318 raft_consensus.cc:504] T 802130b8e94947758554432795f8f8e1 P 0034a6438c7c475abf22bfc0f92af818 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:32.355202 24318 leader_election.cc:288] T 802130b8e94947758554432795f8f8e1 P 0034a6438c7c475abf22bfc0f92af818 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 217ce9884be4472583704a5d3c829f4e (127.21.143.65:45547), 1bd7e55a8bfd4853985e44f1fb798167 (127.21.143.66:33331)
I20260619 18:42:32.360407 24309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:32.355543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:52861 (local address 127.21.143.65:45547)
0619 18:42:32.355697 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:32.355702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:32.355989 (+ 287us) server_negotiation.cc:407] Connection header received
0619 18:42:32.356199 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:32.356203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:32.356263 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:32.356314 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:32.356751 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:32.357500 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:32.359388 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:32.359773 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:32.359957 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:32.360011 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:32.360206 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 18:42:32.360286 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 18:42:32.360435 24358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:32.355516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:45547 (local address 127.21.143.67:52861)
0619 18:42:32.355890 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:32.355906 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:32.356067 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:32.356473 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:32.356475 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:32.356488 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:32.356639 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:32.356649 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:32.357621 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:32.357624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:32.359067 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:32.359078 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:32.359951 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:32.359955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:32.360072 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:32.360094 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 18:42:32.360193 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 18:42:32.360269 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":115,"thread_start_us":112,"threads_started":1}
I20260619 18:42:32.360569 24306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:32.356027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:52987 (local address 127.21.143.66:33331)
0619 18:42:32.356140 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:32.356144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:32.356298 (+ 154us) server_negotiation.cc:407] Connection header received
0619 18:42:32.356414 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:32.356416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:32.356465 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:32.356547 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:32.356838 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:32.357310 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:32.359177 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:32.359811 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:32.359942 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:32.360005 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:32.360376 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 18:42:32.360448 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 18:42:32.360843 23962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "802130b8e94947758554432795f8f8e1" candidate_uuid: "0034a6438c7c475abf22bfc0f92af818" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "217ce9884be4472583704a5d3c829f4e" is_pre_election: true
I20260619 18:42:32.360994 23962 raft_consensus.cc:2454] T 802130b8e94947758554432795f8f8e1 P 217ce9884be4472583704a5d3c829f4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0034a6438c7c475abf22bfc0f92af818 in term 0.
I20260619 18:42:32.361357 24359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:32.355909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:33331 (local address 127.21.143.67:52987)
0619 18:42:32.356216 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:32.356230 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:32.356311 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:32.356577 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:32.356579 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:32.356591 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:32.356683 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:32.356691 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:32.357436 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:32.357439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:32.359067 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:32.359078 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:32.360091 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:32.360095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:32.360238 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:32.360262 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 18:42:32.361131 (+ 869us) client_negotiation.cc:240] Negotiation successful
0619 18:42:32.361205 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":153,"thread_start_us":66,"threads_started":1}
I20260619 18:42:32.361362 24173 leader_election.cc:302] T 802130b8e94947758554432795f8f8e1 P 0034a6438c7c475abf22bfc0f92af818 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0034a6438c7c475abf22bfc0f92af818, 217ce9884be4472583704a5d3c829f4e; no voters:
I20260619 18:42:32.361704 24318 raft_consensus.cc:2781] T 802130b8e94947758554432795f8f8e1 P 0034a6438c7c475abf22bfc0f92af818 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:42:32.361769 24318 raft_consensus.cc:482] T 802130b8e94947758554432795f8f8e1 P 0034a6438c7c475abf22bfc0f92af818 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:42:32.361810 24318 raft_consensus.cc:3037] T 802130b8e94947758554432795f8f8e1 P 0034a6438c7c475abf22bfc0f92af818 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:32.362102 24099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "802130b8e94947758554432795f8f8e1" candidate_uuid: "0034a6438c7c475abf22bfc0f92af818" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bd7e55a8bfd4853985e44f1fb798167" is_pre_election: true
I20260619 18:42:32.362313 24099 raft_consensus.cc:2454] T 802130b8e94947758554432795f8f8e1 P 1bd7e55a8bfd4853985e44f1fb798167 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0034a6438c7c475abf22bfc0f92af818 in term 0.
I20260619 18:42:32.363087 24318 raft_consensus.cc:504] T 802130b8e94947758554432795f8f8e1 P 0034a6438c7c475abf22bfc0f92af818 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:32.363425 24318 leader_election.cc:288] T 802130b8e94947758554432795f8f8e1 P 0034a6438c7c475abf22bfc0f92af818 [CANDIDATE]: Term 1 election: Requested vote from peers 217ce9884be4472583704a5d3c829f4e (127.21.143.65:45547), 1bd7e55a8bfd4853985e44f1fb798167 (127.21.143.66:33331)
I20260619 18:42:32.363693 23962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "802130b8e94947758554432795f8f8e1" candidate_uuid: "0034a6438c7c475abf22bfc0f92af818" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "217ce9884be4472583704a5d3c829f4e"
I20260619 18:42:32.363754 24099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "802130b8e94947758554432795f8f8e1" candidate_uuid: "0034a6438c7c475abf22bfc0f92af818" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bd7e55a8bfd4853985e44f1fb798167"
I20260619 18:42:32.363832 23962 raft_consensus.cc:3037] T 802130b8e94947758554432795f8f8e1 P 217ce9884be4472583704a5d3c829f4e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:32.363897 24099 raft_consensus.cc:3037] T 802130b8e94947758554432795f8f8e1 P 1bd7e55a8bfd4853985e44f1fb798167 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:32.364763 24099 raft_consensus.cc:2454] T 802130b8e94947758554432795f8f8e1 P 1bd7e55a8bfd4853985e44f1fb798167 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0034a6438c7c475abf22bfc0f92af818 in term 1.
I20260619 18:42:32.364909 23962 raft_consensus.cc:2454] T 802130b8e94947758554432795f8f8e1 P 217ce9884be4472583704a5d3c829f4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0034a6438c7c475abf22bfc0f92af818 in term 1.
I20260619 18:42:32.365044 24173 leader_election.cc:302] T 802130b8e94947758554432795f8f8e1 P 0034a6438c7c475abf22bfc0f92af818 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0034a6438c7c475abf22bfc0f92af818, 1bd7e55a8bfd4853985e44f1fb798167; no voters:
I20260619 18:42:32.365190 24318 raft_consensus.cc:2781] T 802130b8e94947758554432795f8f8e1 P 0034a6438c7c475abf22bfc0f92af818 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:32.365362 24318 raft_consensus.cc:686] T 802130b8e94947758554432795f8f8e1 P 0034a6438c7c475abf22bfc0f92af818 [term 1 LEADER]: Becoming Leader. State: Replica: 0034a6438c7c475abf22bfc0f92af818, State: Running, Role: LEADER
I20260619 18:42:32.365592 24318 consensus_queue.cc:237] T 802130b8e94947758554432795f8f8e1 P 0034a6438c7c475abf22bfc0f92af818 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:32.368259 23803 catalog_manager.cc:5495] T 802130b8e94947758554432795f8f8e1 P 0034a6438c7c475abf22bfc0f92af818 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0034a6438c7c475abf22bfc0f92af818 (127.21.143.67). New cstate: current_term: 1 leader_uuid: "0034a6438c7c475abf22bfc0f92af818" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } health_report { overall_health: HEALTHY } } }
I20260619 18:42:32.407013 24310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:32.403165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51644 (local address 127.21.143.67:46073)
0619 18:42:32.403282 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:32.403287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:32.403361 (+ 74us) server_negotiation.cc:407] Connection header received
0619 18:42:32.403533 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:32.403535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:32.403585 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:32.403673 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:42:32.403908 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:32.404391 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:32.405555 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:32.405766 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:32.405876 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:32.406120 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:42:32.406581 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:42:32.406668 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:32.406771 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 18:42:32.406814 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 18:42:32.418051 24306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:32.414550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:51431 (local address 127.21.143.66:33331)
0619 18:42:32.414667 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:32.414673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:32.414866 (+ 193us) server_negotiation.cc:407] Connection header received
0619 18:42:32.414912 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:32.414915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:32.414956 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:32.415019 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:42:32.415276 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:32.415824 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:32.417052 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:32.417310 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:32.417425 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:32.417692 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:42:32.417755 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:42:32.417810 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:32.417908 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 18:42:32.417947 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 18:42:32.418147 24364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:32.414417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:33331 (local address 127.21.143.67:51431)
0619 18:42:32.414730 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:32.414744 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:32.414844 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:32.415053 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:32.415056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:32.415063 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:42:32.415172 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:32.415178 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:32.415965 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:32.415968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:32.416924 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:32.416933 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:32.417427 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:32.417430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:32.417526 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:32.417572 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:42:32.417838 (+ 266us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:42:32.417844 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 18:42:32.417903 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 18:42:32.417981 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":128,"threads_started":1}
I20260619 18:42:32.424459 23962 raft_consensus.cc:1261] T 802130b8e94947758554432795f8f8e1 P 217ce9884be4472583704a5d3c829f4e [term 1 FOLLOWER]: Refusing update from remote peer 0034a6438c7c475abf22bfc0f92af818: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:42:32.424577 24099 raft_consensus.cc:1261] T 802130b8e94947758554432795f8f8e1 P 1bd7e55a8bfd4853985e44f1fb798167 [term 1 FOLLOWER]: Refusing update from remote peer 0034a6438c7c475abf22bfc0f92af818: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:42:32.424952 24360 consensus_queue.cc:1035] T 802130b8e94947758554432795f8f8e1 P 0034a6438c7c475abf22bfc0f92af818 [LEADER]: Connected to new peer: Peer: permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:42:32.425498 24360 consensus_queue.cc:1035] T 802130b8e94947758554432795f8f8e1 P 0034a6438c7c475abf22bfc0f92af818 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:42:32.443877 24310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:32.440503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:37127 (local address 127.21.143.67:46073)
0619 18:42:32.440631 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:32.440635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:32.440646 (+ 11us) server_negotiation.cc:407] Connection header received
0619 18:42:32.440686 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:32.440689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:32.440730 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:32.440825 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:42:32.441069 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:32.441756 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:32.442677 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:32.442949 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:32.443063 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:32.443407 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:42:32.443509 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:42:32.443590 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:32.443693 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 18:42:32.443747 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 18:42:32.443888 24348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:32.440053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:46073 (local address 127.21.143.66:37127)
0619 18:42:32.440279 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:32.440296 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:32.440373 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:32.440835 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:32.440838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:32.440846 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:42:32.440949 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:32.440955 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:32.441883 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:32.441888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:32.442563 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:32.442572 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:32.443068 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:32.443072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:32.443197 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:32.443232 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:42:32.443594 (+ 362us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:42:32.443603 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 18:42:32.443668 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 18:42:32.443750 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
W20260619 18:42:32.444583 24337 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 18:42:34.002347 23803 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:42:34.002521 23803 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:42:34.004665 23803 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 299c0445f59a43ad86d5be47c7d9a35c: Sending DeleteTablet for 3 replicas of tablet 802130b8e94947758554432795f8f8e1
I20260619 18:42:34.005165 23942 tablet_service.cc:1514] Processing DeleteTablet for tablet 802130b8e94947758554432795f8f8e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:42:33 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:36616
I20260619 18:42:34.005223 24082 tablet_service.cc:1514] Processing DeleteTablet for tablet 802130b8e94947758554432795f8f8e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:42:33 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:53296
I20260619 18:42:34.005251 24222 tablet_service.cc:1514] Processing DeleteTablet for tablet 802130b8e94947758554432795f8f8e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:42:33 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:51634
I20260619 18:42:34.005496 24389 tablet_replica.cc:330] T 802130b8e94947758554432795f8f8e1 P 217ce9884be4472583704a5d3c829f4e: stopping tablet replica
I20260619 18:42:34.005749 24389 raft_consensus.cc:2229] T 802130b8e94947758554432795f8f8e1 P 217ce9884be4472583704a5d3c829f4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:42:34.005858 24390 tablet_replica.cc:330] T 802130b8e94947758554432795f8f8e1 P 1bd7e55a8bfd4853985e44f1fb798167: stopping tablet replica
I20260619 18:42:34.005873 24389 raft_consensus.cc:2258] T 802130b8e94947758554432795f8f8e1 P 217ce9884be4472583704a5d3c829f4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:42:34.006067 24390 raft_consensus.cc:2229] T 802130b8e94947758554432795f8f8e1 P 1bd7e55a8bfd4853985e44f1fb798167 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:42:34.006237 24390 raft_consensus.cc:2258] T 802130b8e94947758554432795f8f8e1 P 1bd7e55a8bfd4853985e44f1fb798167 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:42:34.006479 24389 ts_tablet_manager.cc:1905] T 802130b8e94947758554432795f8f8e1 P 217ce9884be4472583704a5d3c829f4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:42:34.006916 24391 tablet_replica.cc:330] T 802130b8e94947758554432795f8f8e1 P 0034a6438c7c475abf22bfc0f92af818: stopping tablet replica
I20260619 18:42:34.007045 24391 raft_consensus.cc:2229] T 802130b8e94947758554432795f8f8e1 P 0034a6438c7c475abf22bfc0f92af818 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:42:34.007351 24391 raft_consensus.cc:2258] T 802130b8e94947758554432795f8f8e1 P 0034a6438c7c475abf22bfc0f92af818 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:42:34.007800 24390 ts_tablet_manager.cc:1905] T 802130b8e94947758554432795f8f8e1 P 1bd7e55a8bfd4853985e44f1fb798167: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:42:34.007918 24391 ts_tablet_manager.cc:1905] T 802130b8e94947758554432795f8f8e1 P 0034a6438c7c475abf22bfc0f92af818: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:42:34.009148 24389 ts_tablet_manager.cc:1918] T 802130b8e94947758554432795f8f8e1 P 217ce9884be4472583704a5d3c829f4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 18:42:34.009230 24389 log.cc:1196] T 802130b8e94947758554432795f8f8e1 P 217ce9884be4472583704a5d3c829f4e: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-0/wal/wals/802130b8e94947758554432795f8f8e1
I20260619 18:42:34.009501 24389 ts_tablet_manager.cc:1939] T 802130b8e94947758554432795f8f8e1 P 217ce9884be4472583704a5d3c829f4e: Deleting consensus metadata
I20260619 18:42:34.010171 23789 catalog_manager.cc:4841] TS 217ce9884be4472583704a5d3c829f4e (127.21.143.65:45547): tablet 802130b8e94947758554432795f8f8e1 (table test-table [id=7ecad7dd7f3c4b5c88a0b548f26452aa]) successfully deleted
I20260619 18:42:34.010612 24391 ts_tablet_manager.cc:1918] T 802130b8e94947758554432795f8f8e1 P 0034a6438c7c475abf22bfc0f92af818: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 18:42:34.010699 24391 log.cc:1196] T 802130b8e94947758554432795f8f8e1 P 0034a6438c7c475abf22bfc0f92af818: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-2/wal/wals/802130b8e94947758554432795f8f8e1
I20260619 18:42:34.010936 24391 ts_tablet_manager.cc:1939] T 802130b8e94947758554432795f8f8e1 P 0034a6438c7c475abf22bfc0f92af818: Deleting consensus metadata
I20260619 18:42:34.011363 24390 ts_tablet_manager.cc:1918] T 802130b8e94947758554432795f8f8e1 P 1bd7e55a8bfd4853985e44f1fb798167: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 18:42:34.011445 24390 log.cc:1196] T 802130b8e94947758554432795f8f8e1 P 1bd7e55a8bfd4853985e44f1fb798167: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-1/wal/wals/802130b8e94947758554432795f8f8e1
I20260619 18:42:34.011569 23789 catalog_manager.cc:4841] TS 0034a6438c7c475abf22bfc0f92af818 (127.21.143.67:46073): tablet 802130b8e94947758554432795f8f8e1 (table test-table [id=7ecad7dd7f3c4b5c88a0b548f26452aa]) successfully deleted
I20260619 18:42:34.011734 24390 ts_tablet_manager.cc:1939] T 802130b8e94947758554432795f8f8e1 P 1bd7e55a8bfd4853985e44f1fb798167: Deleting consensus metadata
I20260619 18:42:34.012241 23789 catalog_manager.cc:4841] TS 1bd7e55a8bfd4853985e44f1fb798167 (127.21.143.66:33331): tablet 802130b8e94947758554432795f8f8e1 (table test-table [id=7ecad7dd7f3c4b5c88a0b548f26452aa]) successfully deleted
Jun 19 18:42:34 dist-test-slave-mld5 krb5kdc[23760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894554, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 18:42:34 dist-test-slave-mld5 krb5kdc[23760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894554, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:34.035132 24399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:34.027173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59104 (local address 127.21.143.126:45255)
0619 18:42:34.027579 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:34.027588 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:34.027604 (+ 16us) server_negotiation.cc:407] Connection header received
0619 18:42:34.027696 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:34.027699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:34.027754 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:34.027833 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:34.028481 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:34.029023 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:34.029782 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:34.029991 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:34.030058 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:34.032379 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:34.032382 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:34.032384 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:34.032405 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:34.033863 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:34.034223 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:34.034225 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:34.034227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:34.034298 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:34.034509 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:34.034511 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:34.034512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:34.034669 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:34.034771 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:34.034916 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 18:42:34.035023 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":249,"threads_started":1}
I20260619 18:42:34.037422 23803 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:42:34.037783 23803 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:42:34.042423 24082 tablet_service.cc:1467] Processing CreateTablet for tablet f94caed8dbf84eeeaabff76f411d7088 (DEFAULT_TABLE table=test-table [id=dc76211ad3bc43c4b0798857cf7c5146]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:42:34.042436 23942 tablet_service.cc:1467] Processing CreateTablet for tablet f94caed8dbf84eeeaabff76f411d7088 (DEFAULT_TABLE table=test-table [id=dc76211ad3bc43c4b0798857cf7c5146]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:42:34.042776 24082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f94caed8dbf84eeeaabff76f411d7088. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:34.042781 23942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f94caed8dbf84eeeaabff76f411d7088. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:34.044853 24400 tablet_bootstrap.cc:492] T f94caed8dbf84eeeaabff76f411d7088 P 217ce9884be4472583704a5d3c829f4e: Bootstrap starting.
I20260619 18:42:34.045083 24401 tablet_bootstrap.cc:492] T f94caed8dbf84eeeaabff76f411d7088 P 1bd7e55a8bfd4853985e44f1fb798167: Bootstrap starting.
I20260619 18:42:34.045259 24222 tablet_service.cc:1467] Processing CreateTablet for tablet f94caed8dbf84eeeaabff76f411d7088 (DEFAULT_TABLE table=test-table [id=dc76211ad3bc43c4b0798857cf7c5146]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:42:34.045526 24222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f94caed8dbf84eeeaabff76f411d7088. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:34.045991 24401 tablet_bootstrap.cc:654] T f94caed8dbf84eeeaabff76f411d7088 P 1bd7e55a8bfd4853985e44f1fb798167: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:34.046144 24400 tablet_bootstrap.cc:654] T f94caed8dbf84eeeaabff76f411d7088 P 217ce9884be4472583704a5d3c829f4e: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:34.047133 24401 tablet_bootstrap.cc:492] T f94caed8dbf84eeeaabff76f411d7088 P 1bd7e55a8bfd4853985e44f1fb798167: No bootstrap required, opened a new log
I20260619 18:42:34.047240 24401 ts_tablet_manager.cc:1397] T f94caed8dbf84eeeaabff76f411d7088 P 1bd7e55a8bfd4853985e44f1fb798167: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 18:42:34.047734 24401 raft_consensus.cc:348] T f94caed8dbf84eeeaabff76f411d7088 P 1bd7e55a8bfd4853985e44f1fb798167 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:34.047869 24401 raft_consensus.cc:374] T f94caed8dbf84eeeaabff76f411d7088 P 1bd7e55a8bfd4853985e44f1fb798167 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:34.047914 24401 raft_consensus.cc:729] T f94caed8dbf84eeeaabff76f411d7088 P 1bd7e55a8bfd4853985e44f1fb798167 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bd7e55a8bfd4853985e44f1fb798167, State: Initialized, Role: FOLLOWER
I20260619 18:42:34.047835 24400 tablet_bootstrap.cc:492] T f94caed8dbf84eeeaabff76f411d7088 P 217ce9884be4472583704a5d3c829f4e: No bootstrap required, opened a new log
I20260619 18:42:34.048043 24401 consensus_queue.cc:260] T f94caed8dbf84eeeaabff76f411d7088 P 1bd7e55a8bfd4853985e44f1fb798167 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:34.048135 24400 ts_tablet_manager.cc:1397] T f94caed8dbf84eeeaabff76f411d7088 P 217ce9884be4472583704a5d3c829f4e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 18:42:34.048369 24401 ts_tablet_manager.cc:1428] T f94caed8dbf84eeeaabff76f411d7088 P 1bd7e55a8bfd4853985e44f1fb798167: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:42:34.048749 24400 raft_consensus.cc:348] T f94caed8dbf84eeeaabff76f411d7088 P 217ce9884be4472583704a5d3c829f4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:34.048866 24400 raft_consensus.cc:374] T f94caed8dbf84eeeaabff76f411d7088 P 217ce9884be4472583704a5d3c829f4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:34.048908 24400 raft_consensus.cc:729] T f94caed8dbf84eeeaabff76f411d7088 P 217ce9884be4472583704a5d3c829f4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 217ce9884be4472583704a5d3c829f4e, State: Initialized, Role: FOLLOWER
I20260619 18:42:34.049026 24400 consensus_queue.cc:260] T f94caed8dbf84eeeaabff76f411d7088 P 217ce9884be4472583704a5d3c829f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:34.049279 24404 tablet_bootstrap.cc:492] T f94caed8dbf84eeeaabff76f411d7088 P 0034a6438c7c475abf22bfc0f92af818: Bootstrap starting.
I20260619 18:42:34.049399 24400 ts_tablet_manager.cc:1428] T f94caed8dbf84eeeaabff76f411d7088 P 217ce9884be4472583704a5d3c829f4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:42:34.050208 24404 tablet_bootstrap.cc:654] T f94caed8dbf84eeeaabff76f411d7088 P 0034a6438c7c475abf22bfc0f92af818: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:34.051522 24404 tablet_bootstrap.cc:492] T f94caed8dbf84eeeaabff76f411d7088 P 0034a6438c7c475abf22bfc0f92af818: No bootstrap required, opened a new log
I20260619 18:42:34.051615 24404 ts_tablet_manager.cc:1397] T f94caed8dbf84eeeaabff76f411d7088 P 0034a6438c7c475abf22bfc0f92af818: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 18:42:34.052024 24404 raft_consensus.cc:348] T f94caed8dbf84eeeaabff76f411d7088 P 0034a6438c7c475abf22bfc0f92af818 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:34.052135 24404 raft_consensus.cc:374] T f94caed8dbf84eeeaabff76f411d7088 P 0034a6438c7c475abf22bfc0f92af818 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:34.052160 24404 raft_consensus.cc:729] T f94caed8dbf84eeeaabff76f411d7088 P 0034a6438c7c475abf22bfc0f92af818 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0034a6438c7c475abf22bfc0f92af818, State: Initialized, Role: FOLLOWER
I20260619 18:42:34.052287 24404 consensus_queue.cc:260] T f94caed8dbf84eeeaabff76f411d7088 P 0034a6438c7c475abf22bfc0f92af818 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:34.052561 24404 ts_tablet_manager.cc:1428] T f94caed8dbf84eeeaabff76f411d7088 P 0034a6438c7c475abf22bfc0f92af818: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 18:42:34.155570 24405 raft_consensus.cc:482] T f94caed8dbf84eeeaabff76f411d7088 P 217ce9884be4472583704a5d3c829f4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:42:34.155735 24405 raft_consensus.cc:504] T f94caed8dbf84eeeaabff76f411d7088 P 217ce9884be4472583704a5d3c829f4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:34.156821 24405 leader_election.cc:288] T f94caed8dbf84eeeaabff76f411d7088 P 217ce9884be4472583704a5d3c829f4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1bd7e55a8bfd4853985e44f1fb798167 (127.21.143.66:33331), 0034a6438c7c475abf22bfc0f92af818 (127.21.143.67:46073)
I20260619 18:42:34.161026 24407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:34.157135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:33331 (local address 127.21.143.65:48325)
0619 18:42:34.157446 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:34.157464 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:34.157580 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:34.157899 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:34.157902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:34.157944 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:34.158068 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:34.158074 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:34.158921 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:34.158927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:34.159930 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:34.159939 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:34.160593 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:34.160596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:34.160692 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:34.160718 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 18:42:34.160801 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 18:42:34.160869 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":18,"spinlock_wait_cycles":24704,"thread_start_us":141,"threads_started":1}
I20260619 18:42:34.161046 24408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:34.157237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:48325 (local address 127.21.143.66:33331)
0619 18:42:34.157543 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:34.157551 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:34.157569 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:42:34.157685 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:34.157688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:34.157776 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:34.157886 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:34.158284 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:34.158809 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:34.160044 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:34.160482 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:34.160605 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:34.160660 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:34.160816 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 18:42:34.160930 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":132,"threads_started":1}
I20260619 18:42:34.161463 24409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:34.157558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:46073 (local address 127.21.143.65:57229)
0619 18:42:34.157826 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:34.157842 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:34.157952 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:34.158227 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:34.158230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:34.158246 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:34.158368 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:34.158373 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:34.159115 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:34.159119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:34.160089 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:34.160098 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:34.160894 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:34.160897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:34.160991 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:34.161017 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 18:42:34.161253 (+ 236us) client_negotiation.cc:240] Negotiation successful
0619 18:42:34.161316 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":18,"spinlock_wait_cycles":3200,"thread_start_us":90,"threads_started":1}
I20260619 18:42:34.161463 24099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f94caed8dbf84eeeaabff76f411d7088" candidate_uuid: "217ce9884be4472583704a5d3c829f4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bd7e55a8bfd4853985e44f1fb798167" is_pre_election: true
I20260619 18:42:34.161603 24099 raft_consensus.cc:2454] T f94caed8dbf84eeeaabff76f411d7088 P 1bd7e55a8bfd4853985e44f1fb798167 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 217ce9884be4472583704a5d3c829f4e in term 0.
I20260619 18:42:34.161676 24410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:34.157670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:57229 (local address 127.21.143.67:46073)
0619 18:42:34.157984 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:34.157995 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:34.158010 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:42:34.158073 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:34.158075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:34.158111 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:34.158203 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:34.158478 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:34.158978 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:34.160203 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:34.160785 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:34.160903 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:34.160948 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:34.161455 (+ 507us) server_negotiation.cc:299] Negotiation successful
0619 18:42:34.161540 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":122,"threads_started":1}
I20260619 18:42:34.162088 23893 leader_election.cc:302] T f94caed8dbf84eeeaabff76f411d7088 P 217ce9884be4472583704a5d3c829f4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bd7e55a8bfd4853985e44f1fb798167, 217ce9884be4472583704a5d3c829f4e; no voters:
I20260619 18:42:34.162184 24242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f94caed8dbf84eeeaabff76f411d7088" candidate_uuid: "217ce9884be4472583704a5d3c829f4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0034a6438c7c475abf22bfc0f92af818" is_pre_election: true
I20260619 18:42:34.162328 24242 raft_consensus.cc:2454] T f94caed8dbf84eeeaabff76f411d7088 P 0034a6438c7c475abf22bfc0f92af818 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 217ce9884be4472583704a5d3c829f4e in term 0.
I20260619 18:42:34.162418 24405 raft_consensus.cc:2781] T f94caed8dbf84eeeaabff76f411d7088 P 217ce9884be4472583704a5d3c829f4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:42:34.162479 24405 raft_consensus.cc:482] T f94caed8dbf84eeeaabff76f411d7088 P 217ce9884be4472583704a5d3c829f4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:42:34.162521 24405 raft_consensus.cc:3037] T f94caed8dbf84eeeaabff76f411d7088 P 217ce9884be4472583704a5d3c829f4e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:34.163549 24405 raft_consensus.cc:504] T f94caed8dbf84eeeaabff76f411d7088 P 217ce9884be4472583704a5d3c829f4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:34.163863 24405 leader_election.cc:288] T f94caed8dbf84eeeaabff76f411d7088 P 217ce9884be4472583704a5d3c829f4e [CANDIDATE]: Term 1 election: Requested vote from peers 1bd7e55a8bfd4853985e44f1fb798167 (127.21.143.66:33331), 0034a6438c7c475abf22bfc0f92af818 (127.21.143.67:46073)
I20260619 18:42:34.164191 24099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f94caed8dbf84eeeaabff76f411d7088" candidate_uuid: "217ce9884be4472583704a5d3c829f4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bd7e55a8bfd4853985e44f1fb798167"
I20260619 18:42:34.164247 24242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f94caed8dbf84eeeaabff76f411d7088" candidate_uuid: "217ce9884be4472583704a5d3c829f4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0034a6438c7c475abf22bfc0f92af818"
I20260619 18:42:34.164314 24099 raft_consensus.cc:3037] T f94caed8dbf84eeeaabff76f411d7088 P 1bd7e55a8bfd4853985e44f1fb798167 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:34.164356 24242 raft_consensus.cc:3037] T f94caed8dbf84eeeaabff76f411d7088 P 0034a6438c7c475abf22bfc0f92af818 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:34.165198 24099 raft_consensus.cc:2454] T f94caed8dbf84eeeaabff76f411d7088 P 1bd7e55a8bfd4853985e44f1fb798167 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 217ce9884be4472583704a5d3c829f4e in term 1.
I20260619 18:42:34.165202 24242 raft_consensus.cc:2454] T f94caed8dbf84eeeaabff76f411d7088 P 0034a6438c7c475abf22bfc0f92af818 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 217ce9884be4472583704a5d3c829f4e in term 1.
I20260619 18:42:34.165470 23893 leader_election.cc:302] T f94caed8dbf84eeeaabff76f411d7088 P 217ce9884be4472583704a5d3c829f4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0034a6438c7c475abf22bfc0f92af818, 217ce9884be4472583704a5d3c829f4e; no voters:
I20260619 18:42:34.165644 24405 raft_consensus.cc:2781] T f94caed8dbf84eeeaabff76f411d7088 P 217ce9884be4472583704a5d3c829f4e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:34.165866 24405 raft_consensus.cc:686] T f94caed8dbf84eeeaabff76f411d7088 P 217ce9884be4472583704a5d3c829f4e [term 1 LEADER]: Becoming Leader. State: Replica: 217ce9884be4472583704a5d3c829f4e, State: Running, Role: LEADER
I20260619 18:42:34.166126 24405 consensus_queue.cc:237] T f94caed8dbf84eeeaabff76f411d7088 P 217ce9884be4472583704a5d3c829f4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:34.168391 23804 catalog_manager.cc:5495] T f94caed8dbf84eeeaabff76f411d7088 P 217ce9884be4472583704a5d3c829f4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 217ce9884be4472583704a5d3c829f4e (127.21.143.65). New cstate: current_term: 1 leader_uuid: "217ce9884be4472583704a5d3c829f4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } health_report { overall_health: UNKNOWN } } }
I20260619 18:42:34.205120 24414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:34.201098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36632 (local address 127.21.143.65:45547)
0619 18:42:34.201431 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:34.201440 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:34.201454 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:42:34.201586 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:34.201588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:34.201661 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:34.201789 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:42:34.202057 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:34.202609 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:34.203526 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:34.203787 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:34.203903 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:34.204200 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:42:34.204684 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:42:34.204821 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:34.204883 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 18:42:34.204937 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":97,"threads_started":1}
I20260619 18:42:34.215248 24408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:34.212314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:60839 (local address 127.21.143.66:33331)
0619 18:42:34.212450 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:34.212453 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:34.212539 (+ 86us) server_negotiation.cc:407] Connection header received
0619 18:42:34.212651 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:34.212653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:34.212698 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:34.212766 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:42:34.212985 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:34.213517 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:34.214335 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:34.214551 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:34.214649 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:34.214901 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:42:34.214964 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:42:34.215017 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:34.215095 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 18:42:34.215141 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 18:42:34.215263 24417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:34.212124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:33331 (local address 127.21.143.65:60839)
0619 18:42:34.212460 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:34.212474 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:34.212557 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:34.212785 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:34.212787 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:34.212792 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:42:34.212895 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:34.212899 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:34.213602 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:34.213604 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:34.214210 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:34.214218 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:34.214656 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:34.214658 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:34.214737 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:34.214786 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:42:34.215036 (+ 250us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:42:34.215041 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 18:42:34.215089 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 18:42:34.215147 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":143,"threads_started":1}
I20260619 18:42:34.221446 24099 raft_consensus.cc:1261] T f94caed8dbf84eeeaabff76f411d7088 P 1bd7e55a8bfd4853985e44f1fb798167 [term 1 FOLLOWER]: Refusing update from remote peer 217ce9884be4472583704a5d3c829f4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:42:34.221482 24242 raft_consensus.cc:1261] T f94caed8dbf84eeeaabff76f411d7088 P 0034a6438c7c475abf22bfc0f92af818 [term 1 FOLLOWER]: Refusing update from remote peer 217ce9884be4472583704a5d3c829f4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:42:34.221892 24405 consensus_queue.cc:1035] T f94caed8dbf84eeeaabff76f411d7088 P 217ce9884be4472583704a5d3c829f4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:42:34.222142 24411 consensus_queue.cc:1035] T f94caed8dbf84eeeaabff76f411d7088 P 217ce9884be4472583704a5d3c829f4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:42:34.238947 24414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:34.235689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:48795 (local address 127.21.143.65:45547)
0619 18:42:34.235908 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:34.235915 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:34.235932 (+ 17us) server_negotiation.cc:407] Connection header received
0619 18:42:34.236033 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:34.236036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:34.236090 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:34.236178 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:42:34.236376 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:34.236861 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:34.237808 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:34.238046 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:34.238161 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:34.238412 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:42:34.238495 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:42:34.238577 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:34.238760 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 18:42:34.238814 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260619 18:42:34.238962 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:34.235629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:45547 (local address 127.21.143.66:48795)
0619 18:42:34.235871 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:34.235883 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:34.235957 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:34.236200 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:34.236203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:34.236211 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:42:34.236309 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:34.236314 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:34.236958 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:34.236961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:34.237692 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:34.237701 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:34.238160 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:34.238164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:34.238258 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:34.238289 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:42:34.238619 (+ 330us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:42:34.238630 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 18:42:34.238741 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 18:42:34.238836 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":99,"threads_started":1}
I20260619 18:42:34.260092 23804 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:42:34.260282 23804 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:42:34.262363 23804 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 299c0445f59a43ad86d5be47c7d9a35c: Sending DeleteTablet for 3 replicas of tablet f94caed8dbf84eeeaabff76f411d7088
I20260619 18:42:34.262997 23942 tablet_service.cc:1514] Processing DeleteTablet for tablet f94caed8dbf84eeeaabff76f411d7088 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:42:34 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:36616
I20260619 18:42:34.262984 24082 tablet_service.cc:1514] Processing DeleteTablet for tablet f94caed8dbf84eeeaabff76f411d7088 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:42:34 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:53296
I20260619 18:42:34.263154 24389 tablet_replica.cc:330] T f94caed8dbf84eeeaabff76f411d7088 P 217ce9884be4472583704a5d3c829f4e: stopping tablet replica
I20260619 18:42:34.263154 24390 tablet_replica.cc:330] T f94caed8dbf84eeeaabff76f411d7088 P 1bd7e55a8bfd4853985e44f1fb798167: stopping tablet replica
I20260619 18:42:34.263253 24389 raft_consensus.cc:2229] T f94caed8dbf84eeeaabff76f411d7088 P 217ce9884be4472583704a5d3c829f4e [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:42:34.263253 24390 raft_consensus.cc:2229] T f94caed8dbf84eeeaabff76f411d7088 P 1bd7e55a8bfd4853985e44f1fb798167 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:42:34.263372 24390 raft_consensus.cc:2258] T f94caed8dbf84eeeaabff76f411d7088 P 1bd7e55a8bfd4853985e44f1fb798167 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:42:34.263551 24389 raft_consensus.cc:2258] T f94caed8dbf84eeeaabff76f411d7088 P 217ce9884be4472583704a5d3c829f4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:42:34.263586 23804 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:42:34.264062 23804 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:42:34.264242 24390 ts_tablet_manager.cc:1905] T f94caed8dbf84eeeaabff76f411d7088 P 1bd7e55a8bfd4853985e44f1fb798167: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:42:34.264971 24222 tablet_service.cc:1514] Processing DeleteTablet for tablet f94caed8dbf84eeeaabff76f411d7088 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:42:34 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:51634
I20260619 18:42:34.267148 24390 ts_tablet_manager.cc:1918] T f94caed8dbf84eeeaabff76f411d7088 P 1bd7e55a8bfd4853985e44f1fb798167: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:42:34.267227 24390 log.cc:1196] T f94caed8dbf84eeeaabff76f411d7088 P 1bd7e55a8bfd4853985e44f1fb798167: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-1/wal/wals/f94caed8dbf84eeeaabff76f411d7088
I20260619 18:42:34.267517 24390 ts_tablet_manager.cc:1939] T f94caed8dbf84eeeaabff76f411d7088 P 1bd7e55a8bfd4853985e44f1fb798167: Deleting consensus metadata
I20260619 18:42:34.268239 24391 tablet_replica.cc:330] T f94caed8dbf84eeeaabff76f411d7088 P 0034a6438c7c475abf22bfc0f92af818: stopping tablet replica
I20260619 18:42:34.268399 24391 raft_consensus.cc:2229] T f94caed8dbf84eeeaabff76f411d7088 P 0034a6438c7c475abf22bfc0f92af818 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:42:34.268527 24391 raft_consensus.cc:2258] T f94caed8dbf84eeeaabff76f411d7088 P 0034a6438c7c475abf22bfc0f92af818 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:42:34.268865 23789 catalog_manager.cc:4841] TS 1bd7e55a8bfd4853985e44f1fb798167 (127.21.143.66:33331): tablet f94caed8dbf84eeeaabff76f411d7088 (table test-table [id=dc76211ad3bc43c4b0798857cf7c5146]) successfully deleted
I20260619 18:42:34.269460 23942 tablet_service.cc:1467] Processing CreateTablet for tablet b43880264fcb4d8f9190fbf2394fa189 (DEFAULT_TABLE table=test-table [id=b680a7eda24c4bf18839c6b7bbbee1fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:42:34.269493 24082 tablet_service.cc:1467] Processing CreateTablet for tablet b43880264fcb4d8f9190fbf2394fa189 (DEFAULT_TABLE table=test-table [id=b680a7eda24c4bf18839c6b7bbbee1fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:42:34.269709 24082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b43880264fcb4d8f9190fbf2394fa189. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:34.269713 23942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b43880264fcb4d8f9190fbf2394fa189. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:34.271905 24401 tablet_bootstrap.cc:492] T b43880264fcb4d8f9190fbf2394fa189 P 1bd7e55a8bfd4853985e44f1fb798167: Bootstrap starting.
I20260619 18:42:34.272064 24400 tablet_bootstrap.cc:492] T b43880264fcb4d8f9190fbf2394fa189 P 217ce9884be4472583704a5d3c829f4e: Bootstrap starting.
I20260619 18:42:34.272514 24391 ts_tablet_manager.cc:1905] T f94caed8dbf84eeeaabff76f411d7088 P 0034a6438c7c475abf22bfc0f92af818: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:42:34.272821 24401 tablet_bootstrap.cc:654] T b43880264fcb4d8f9190fbf2394fa189 P 1bd7e55a8bfd4853985e44f1fb798167: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:34.272957 24400 tablet_bootstrap.cc:654] T b43880264fcb4d8f9190fbf2394fa189 P 217ce9884be4472583704a5d3c829f4e: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:34.274057 24401 tablet_bootstrap.cc:492] T b43880264fcb4d8f9190fbf2394fa189 P 1bd7e55a8bfd4853985e44f1fb798167: No bootstrap required, opened a new log
I20260619 18:42:34.274158 24401 ts_tablet_manager.cc:1397] T b43880264fcb4d8f9190fbf2394fa189 P 1bd7e55a8bfd4853985e44f1fb798167: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 18:42:34.274544 24401 raft_consensus.cc:348] T b43880264fcb4d8f9190fbf2394fa189 P 1bd7e55a8bfd4853985e44f1fb798167 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:34.274662 24401 raft_consensus.cc:374] T b43880264fcb4d8f9190fbf2394fa189 P 1bd7e55a8bfd4853985e44f1fb798167 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:34.274693 24401 raft_consensus.cc:729] T b43880264fcb4d8f9190fbf2394fa189 P 1bd7e55a8bfd4853985e44f1fb798167 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bd7e55a8bfd4853985e44f1fb798167, State: Initialized, Role: FOLLOWER
I20260619 18:42:34.274693 24391 ts_tablet_manager.cc:1918] T f94caed8dbf84eeeaabff76f411d7088 P 0034a6438c7c475abf22bfc0f92af818: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:42:34.274760 24391 log.cc:1196] T f94caed8dbf84eeeaabff76f411d7088 P 0034a6438c7c475abf22bfc0f92af818: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-2/wal/wals/f94caed8dbf84eeeaabff76f411d7088
I20260619 18:42:34.274811 24401 consensus_queue.cc:260] T b43880264fcb4d8f9190fbf2394fa189 P 1bd7e55a8bfd4853985e44f1fb798167 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:34.275045 24391 ts_tablet_manager.cc:1939] T f94caed8dbf84eeeaabff76f411d7088 P 0034a6438c7c475abf22bfc0f92af818: Deleting consensus metadata
I20260619 18:42:34.275059 24401 ts_tablet_manager.cc:1428] T b43880264fcb4d8f9190fbf2394fa189 P 1bd7e55a8bfd4853985e44f1fb798167: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:42:34.275725 24389 ts_tablet_manager.cc:1905] T f94caed8dbf84eeeaabff76f411d7088 P 217ce9884be4472583704a5d3c829f4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:42:34.276593 24400 tablet_bootstrap.cc:492] T b43880264fcb4d8f9190fbf2394fa189 P 217ce9884be4472583704a5d3c829f4e: No bootstrap required, opened a new log
I20260619 18:42:34.276678 24400 ts_tablet_manager.cc:1397] T b43880264fcb4d8f9190fbf2394fa189 P 217ce9884be4472583704a5d3c829f4e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260619 18:42:34.277107 24400 raft_consensus.cc:348] T b43880264fcb4d8f9190fbf2394fa189 P 217ce9884be4472583704a5d3c829f4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:34.277227 24400 raft_consensus.cc:374] T b43880264fcb4d8f9190fbf2394fa189 P 217ce9884be4472583704a5d3c829f4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:34.277258 24400 raft_consensus.cc:729] T b43880264fcb4d8f9190fbf2394fa189 P 217ce9884be4472583704a5d3c829f4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 217ce9884be4472583704a5d3c829f4e, State: Initialized, Role: FOLLOWER
I20260619 18:42:34.277379 24400 consensus_queue.cc:260] T b43880264fcb4d8f9190fbf2394fa189 P 217ce9884be4472583704a5d3c829f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:34.277639 24400 ts_tablet_manager.cc:1428] T b43880264fcb4d8f9190fbf2394fa189 P 217ce9884be4472583704a5d3c829f4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:42:34.278076 24389 ts_tablet_manager.cc:1918] T f94caed8dbf84eeeaabff76f411d7088 P 217ce9884be4472583704a5d3c829f4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:42:34.278172 24389 log.cc:1196] T f94caed8dbf84eeeaabff76f411d7088 P 217ce9884be4472583704a5d3c829f4e: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-0/wal/wals/f94caed8dbf84eeeaabff76f411d7088
I20260619 18:42:34.278445 24389 ts_tablet_manager.cc:1939] T f94caed8dbf84eeeaabff76f411d7088 P 217ce9884be4472583704a5d3c829f4e: Deleting consensus metadata
I20260619 18:42:34.279367 23789 catalog_manager.cc:4841] TS 217ce9884be4472583704a5d3c829f4e (127.21.143.65:45547): tablet f94caed8dbf84eeeaabff76f411d7088 (table test-table [id=dc76211ad3bc43c4b0798857cf7c5146]) successfully deleted
I20260619 18:42:34.280009 24222 tablet_service.cc:1467] Processing CreateTablet for tablet b43880264fcb4d8f9190fbf2394fa189 (DEFAULT_TABLE table=test-table [id=b680a7eda24c4bf18839c6b7bbbee1fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:42:34.280256 24222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b43880264fcb4d8f9190fbf2394fa189. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:34.280278 23789 catalog_manager.cc:4841] TS 0034a6438c7c475abf22bfc0f92af818 (127.21.143.67:46073): tablet f94caed8dbf84eeeaabff76f411d7088 (table test-table [id=dc76211ad3bc43c4b0798857cf7c5146]) successfully deleted
I20260619 18:42:34.282101 24404 tablet_bootstrap.cc:492] T b43880264fcb4d8f9190fbf2394fa189 P 0034a6438c7c475abf22bfc0f92af818: Bootstrap starting.
I20260619 18:42:34.282862 24404 tablet_bootstrap.cc:654] T b43880264fcb4d8f9190fbf2394fa189 P 0034a6438c7c475abf22bfc0f92af818: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:34.286648 24404 tablet_bootstrap.cc:492] T b43880264fcb4d8f9190fbf2394fa189 P 0034a6438c7c475abf22bfc0f92af818: No bootstrap required, opened a new log
I20260619 18:42:34.286748 24404 ts_tablet_manager.cc:1397] T b43880264fcb4d8f9190fbf2394fa189 P 0034a6438c7c475abf22bfc0f92af818: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260619 18:42:34.287075 24404 raft_consensus.cc:348] T b43880264fcb4d8f9190fbf2394fa189 P 0034a6438c7c475abf22bfc0f92af818 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:34.287192 24404 raft_consensus.cc:374] T b43880264fcb4d8f9190fbf2394fa189 P 0034a6438c7c475abf22bfc0f92af818 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:34.287238 24404 raft_consensus.cc:729] T b43880264fcb4d8f9190fbf2394fa189 P 0034a6438c7c475abf22bfc0f92af818 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0034a6438c7c475abf22bfc0f92af818, State: Initialized, Role: FOLLOWER
I20260619 18:42:34.287396 24404 consensus_queue.cc:260] T b43880264fcb4d8f9190fbf2394fa189 P 0034a6438c7c475abf22bfc0f92af818 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:34.287705 24404 ts_tablet_manager.cc:1428] T b43880264fcb4d8f9190fbf2394fa189 P 0034a6438c7c475abf22bfc0f92af818: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:42:34.306701 24422 raft_consensus.cc:482] T b43880264fcb4d8f9190fbf2394fa189 P 217ce9884be4472583704a5d3c829f4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:42:34.306828 24422 raft_consensus.cc:504] T b43880264fcb4d8f9190fbf2394fa189 P 217ce9884be4472583704a5d3c829f4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:34.307255 24422 leader_election.cc:288] T b43880264fcb4d8f9190fbf2394fa189 P 217ce9884be4472583704a5d3c829f4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1bd7e55a8bfd4853985e44f1fb798167 (127.21.143.66:33331), 0034a6438c7c475abf22bfc0f92af818 (127.21.143.67:46073)
I20260619 18:42:34.307626 24099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b43880264fcb4d8f9190fbf2394fa189" candidate_uuid: "217ce9884be4472583704a5d3c829f4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bd7e55a8bfd4853985e44f1fb798167" is_pre_election: true
I20260619 18:42:34.307760 24099 raft_consensus.cc:2454] T b43880264fcb4d8f9190fbf2394fa189 P 1bd7e55a8bfd4853985e44f1fb798167 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 217ce9884be4472583704a5d3c829f4e in term 0.
I20260619 18:42:34.307976 24242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b43880264fcb4d8f9190fbf2394fa189" candidate_uuid: "217ce9884be4472583704a5d3c829f4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0034a6438c7c475abf22bfc0f92af818" is_pre_election: true
I20260619 18:42:34.308059 23893 leader_election.cc:302] T b43880264fcb4d8f9190fbf2394fa189 P 217ce9884be4472583704a5d3c829f4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bd7e55a8bfd4853985e44f1fb798167, 217ce9884be4472583704a5d3c829f4e; no voters:
I20260619 18:42:34.308113 24242 raft_consensus.cc:2454] T b43880264fcb4d8f9190fbf2394fa189 P 0034a6438c7c475abf22bfc0f92af818 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 217ce9884be4472583704a5d3c829f4e in term 0.
I20260619 18:42:34.308224 24422 raft_consensus.cc:2781] T b43880264fcb4d8f9190fbf2394fa189 P 217ce9884be4472583704a5d3c829f4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:42:34.308266 24422 raft_consensus.cc:482] T b43880264fcb4d8f9190fbf2394fa189 P 217ce9884be4472583704a5d3c829f4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:42:34.308301 24422 raft_consensus.cc:3037] T b43880264fcb4d8f9190fbf2394fa189 P 217ce9884be4472583704a5d3c829f4e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:34.309952 24422 raft_consensus.cc:504] T b43880264fcb4d8f9190fbf2394fa189 P 217ce9884be4472583704a5d3c829f4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:34.310276 24422 leader_election.cc:288] T b43880264fcb4d8f9190fbf2394fa189 P 217ce9884be4472583704a5d3c829f4e [CANDIDATE]: Term 1 election: Requested vote from peers 1bd7e55a8bfd4853985e44f1fb798167 (127.21.143.66:33331), 0034a6438c7c475abf22bfc0f92af818 (127.21.143.67:46073)
I20260619 18:42:34.310549 24099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b43880264fcb4d8f9190fbf2394fa189" candidate_uuid: "217ce9884be4472583704a5d3c829f4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bd7e55a8bfd4853985e44f1fb798167"
I20260619 18:42:34.310658 24099 raft_consensus.cc:3037] T b43880264fcb4d8f9190fbf2394fa189 P 1bd7e55a8bfd4853985e44f1fb798167 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:34.310636 24242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b43880264fcb4d8f9190fbf2394fa189" candidate_uuid: "217ce9884be4472583704a5d3c829f4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0034a6438c7c475abf22bfc0f92af818"
I20260619 18:42:34.310729 24242 raft_consensus.cc:3037] T b43880264fcb4d8f9190fbf2394fa189 P 0034a6438c7c475abf22bfc0f92af818 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:34.311733 24099 raft_consensus.cc:2454] T b43880264fcb4d8f9190fbf2394fa189 P 1bd7e55a8bfd4853985e44f1fb798167 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 217ce9884be4472583704a5d3c829f4e in term 1.
I20260619 18:42:34.312005 23893 leader_election.cc:302] T b43880264fcb4d8f9190fbf2394fa189 P 217ce9884be4472583704a5d3c829f4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bd7e55a8bfd4853985e44f1fb798167, 217ce9884be4472583704a5d3c829f4e; no voters:
I20260619 18:42:34.312139 24242 raft_consensus.cc:2454] T b43880264fcb4d8f9190fbf2394fa189 P 0034a6438c7c475abf22bfc0f92af818 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 217ce9884be4472583704a5d3c829f4e in term 1.
I20260619 18:42:34.312587 24422 raft_consensus.cc:2781] T b43880264fcb4d8f9190fbf2394fa189 P 217ce9884be4472583704a5d3c829f4e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:34.312923 24422 raft_consensus.cc:686] T b43880264fcb4d8f9190fbf2394fa189 P 217ce9884be4472583704a5d3c829f4e [term 1 LEADER]: Becoming Leader. State: Replica: 217ce9884be4472583704a5d3c829f4e, State: Running, Role: LEADER
I20260619 18:42:34.313066 24422 consensus_queue.cc:237] T b43880264fcb4d8f9190fbf2394fa189 P 217ce9884be4472583704a5d3c829f4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } }
I20260619 18:42:34.314720 23804 catalog_manager.cc:5495] T b43880264fcb4d8f9190fbf2394fa189 P 217ce9884be4472583704a5d3c829f4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 217ce9884be4472583704a5d3c829f4e (127.21.143.65). New cstate: current_term: 1 leader_uuid: "217ce9884be4472583704a5d3c829f4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "217ce9884be4472583704a5d3c829f4e" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 } health_report { overall_health: UNKNOWN } } }
I20260619 18:42:34.336895 24242 raft_consensus.cc:1261] T b43880264fcb4d8f9190fbf2394fa189 P 0034a6438c7c475abf22bfc0f92af818 [term 1 FOLLOWER]: Refusing update from remote peer 217ce9884be4472583704a5d3c829f4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:42:34.336949 24099 raft_consensus.cc:1261] T b43880264fcb4d8f9190fbf2394fa189 P 1bd7e55a8bfd4853985e44f1fb798167 [term 1 FOLLOWER]: Refusing update from remote peer 217ce9884be4472583704a5d3c829f4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:42:34.337330 24422 consensus_queue.cc:1035] T b43880264fcb4d8f9190fbf2394fa189 P 217ce9884be4472583704a5d3c829f4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0034a6438c7c475abf22bfc0f92af818" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 46073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:42:34.337572 24411 consensus_queue.cc:1035] T b43880264fcb4d8f9190fbf2394fa189 P 217ce9884be4472583704a5d3c829f4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bd7e55a8bfd4853985e44f1fb798167" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 18:42:34.352147 24373 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 18:42:35.904978 23803 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:42:35.905131 23803 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:42:35.907574 23803 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 299c0445f59a43ad86d5be47c7d9a35c: Sending DeleteTablet for 3 replicas of tablet b43880264fcb4d8f9190fbf2394fa189
I20260619 18:42:35.908140 23942 tablet_service.cc:1514] Processing DeleteTablet for tablet b43880264fcb4d8f9190fbf2394fa189 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:42:35 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:36616
I20260619 18:42:35.908207 24081 tablet_service.cc:1514] Processing DeleteTablet for tablet b43880264fcb4d8f9190fbf2394fa189 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:42:35 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:53296
I20260619 18:42:35.908326 24222 tablet_service.cc:1514] Processing DeleteTablet for tablet b43880264fcb4d8f9190fbf2394fa189 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:42:35 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:51634
I20260619 18:42:35.908514 24454 tablet_replica.cc:330] T b43880264fcb4d8f9190fbf2394fa189 P 1bd7e55a8bfd4853985e44f1fb798167: stopping tablet replica
I20260619 18:42:35.908514 24453 tablet_replica.cc:330] T b43880264fcb4d8f9190fbf2394fa189 P 217ce9884be4472583704a5d3c829f4e: stopping tablet replica
I20260619 18:42:35.908631 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 23877
I20260619 18:42:35.908638 24453 raft_consensus.cc:2229] T b43880264fcb4d8f9190fbf2394fa189 P 217ce9884be4472583704a5d3c829f4e [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:42:35.908669 24454 raft_consensus.cc:2229] T b43880264fcb4d8f9190fbf2394fa189 P 1bd7e55a8bfd4853985e44f1fb798167 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:42:35.908771 24454 raft_consensus.cc:2258] T b43880264fcb4d8f9190fbf2394fa189 P 1bd7e55a8bfd4853985e44f1fb798167 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:42:35.909353 24454 ts_tablet_manager.cc:1905] T b43880264fcb4d8f9190fbf2394fa189 P 1bd7e55a8bfd4853985e44f1fb798167: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:42:35.911046 24455 tablet_replica.cc:330] T b43880264fcb4d8f9190fbf2394fa189 P 0034a6438c7c475abf22bfc0f92af818: stopping tablet replica
I20260619 18:42:35.911202 24455 raft_consensus.cc:2229] T b43880264fcb4d8f9190fbf2394fa189 P 0034a6438c7c475abf22bfc0f92af818 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:42:35.911355 24455 raft_consensus.cc:2258] T b43880264fcb4d8f9190fbf2394fa189 P 0034a6438c7c475abf22bfc0f92af818 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:42:35.911986 24455 ts_tablet_manager.cc:1905] T b43880264fcb4d8f9190fbf2394fa189 P 0034a6438c7c475abf22bfc0f92af818: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:42:35.912683 24454 ts_tablet_manager.cc:1918] T b43880264fcb4d8f9190fbf2394fa189 P 1bd7e55a8bfd4853985e44f1fb798167: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 18:42:35.912765 24454 log.cc:1196] T b43880264fcb4d8f9190fbf2394fa189 P 1bd7e55a8bfd4853985e44f1fb798167: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-1/wal/wals/b43880264fcb4d8f9190fbf2394fa189
I20260619 18:42:35.913059 24454 ts_tablet_manager.cc:1939] T b43880264fcb4d8f9190fbf2394fa189 P 1bd7e55a8bfd4853985e44f1fb798167: Deleting consensus metadata
I20260619 18:42:35.913858 23789 catalog_manager.cc:4841] TS 1bd7e55a8bfd4853985e44f1fb798167 (127.21.143.66:33331): tablet b43880264fcb4d8f9190fbf2394fa189 (table test-table [id=b680a7eda24c4bf18839c6b7bbbee1fb]) successfully deleted
I20260619 18:42:35.914286 24455 ts_tablet_manager.cc:1918] T b43880264fcb4d8f9190fbf2394fa189 P 0034a6438c7c475abf22bfc0f92af818: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 18:42:35.914373 24455 log.cc:1196] T b43880264fcb4d8f9190fbf2394fa189 P 0034a6438c7c475abf22bfc0f92af818: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894542572984-22077-0/minicluster-data/ts-2/wal/wals/b43880264fcb4d8f9190fbf2394fa189
I20260619 18:42:35.914629 24455 ts_tablet_manager.cc:1939] T b43880264fcb4d8f9190fbf2394fa189 P 0034a6438c7c475abf22bfc0f92af818: Deleting consensus metadata
I20260619 18:42:35.915203 23789 catalog_manager.cc:4841] TS 0034a6438c7c475abf22bfc0f92af818 (127.21.143.67:46073): tablet b43880264fcb4d8f9190fbf2394fa189 (table test-table [id=b680a7eda24c4bf18839c6b7bbbee1fb]) successfully deleted
W20260619 18:42:35.917237 23789 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.143.65:45547 (error 108)
I20260619 18:42:35.917490 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 24017
I20260619 18:42:35.918850 24457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:35.918471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:45547 (local address 127.0.0.1:36642)
0619 18:42:35.918690 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:35.918766 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.65:45547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":89,"threads_started":1}
W20260619 18:42:35.919083 23789 catalog_manager.cc:4584] TS 217ce9884be4472583704a5d3c829f4e (127.21.143.65:45547): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b43880264fcb4d8f9190fbf2394fa189: Network error: Client connection negotiation failed: client connection to 127.21.143.65:45547: connect: Connection refused (error 111)
I20260619 18:42:35.924852 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 24157
I20260619 18:42:35.931556 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 23773
2026-06-19T18:42:35Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6244 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:42:35 dist-test-slave-mld5 krb5kdc[24460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:42:35 dist-test-slave-mld5 krb5kdc[24460](info): set up 2 sockets
Jun 19 18:42:35 dist-test-slave-mld5 krb5kdc[24460](info): commencing operation
krb5kdc: starting...
W20260619 18:42:36.967550 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:42:37 dist-test-slave-mld5 krb5kdc[24460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:42:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:42:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.143.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
I20260619 18:42:37.082017 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:35757
--webserver_interface=127.21.143.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:38709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:35757
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:37.187821 24473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:37.188062 24473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:37.188119 24473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:37.192160 24473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:42:37.192220 24473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:37.192240 24473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:37.192256 24473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:42:37.192272 24473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:42:37.197482 24473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:38709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:35757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:35757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:37.198602 24473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:37.199736 24473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:37.208505 24481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:37.208437 24478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:37.208513 24479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:37.209052 24473 server_base.cc:1034] running on GCE node
I20260619 18:42:37.209715 24473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:37.211119 24473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:37.212337 24473 hybrid_clock.cc:648] HybridClock initialized: now 1781894557212312 us; error 45 us; skew 500 ppm
Jun 19 18:42:37 dist-test-slave-mld5 krb5kdc[24460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894557, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:37.218531 24473 init.cc:377] Logged in from keytab as kudu/127.21.143.126@KRBTEST.COM (short username kudu)
I20260619 18:42:37.219553 24473 webserver.cc:460] Webserver started at http://127.21.143.126:38799/ using document root <none> and password file <none>
I20260619 18:42:37.220104 24473 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:37.220163 24473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:37.220327 24473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:37.222362 24473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "d8b56e0554174991965bef333e942440"
format_stamp: "Formatted at 2026-06-19 18:42:37 on dist-test-slave-mld5"
server_key: "c8e4a451ae6a83ef273ae6ca949edb6e"
server_key_iv: "5fd38fe475296ad3efba077a4222abed"
server_key_version: "encryptionkey@0"
I20260619 18:42:37.222867 24473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "d8b56e0554174991965bef333e942440"
format_stamp: "Formatted at 2026-06-19 18:42:37 on dist-test-slave-mld5"
server_key: "c8e4a451ae6a83ef273ae6ca949edb6e"
server_key_iv: "5fd38fe475296ad3efba077a4222abed"
server_key_version: "encryptionkey@0"
I20260619 18:42:37.226464 24473 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260619 18:42:37.228811 24488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:37.230037 24473 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260619 18:42:37.230144 24473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "d8b56e0554174991965bef333e942440"
format_stamp: "Formatted at 2026-06-19 18:42:37 on dist-test-slave-mld5"
server_key: "c8e4a451ae6a83ef273ae6ca949edb6e"
server_key_iv: "5fd38fe475296ad3efba077a4222abed"
server_key_version: "encryptionkey@0"
I20260619 18:42:37.230224 24473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:37.243672 24473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:37.246798 24473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:37.246953 24473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:37.253794 24473 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:35757
I20260619 18:42:37.253815 24540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:35757 every 8 connection(s)
I20260619 18:42:37.254724 24473 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:42:37.257349 24541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:37.257519 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 24473
I20260619 18:42:37.257632 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:42:37.257927 22077 external_mini_cluster.cc:1428] Setting key e2ce8e7b8440a9c50d10cce0beb4f144
I20260619 18:42:37.263559 24541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8b56e0554174991965bef333e942440: Bootstrap starting.
I20260619 18:42:37.265712 24541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8b56e0554174991965bef333e942440: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:37.266419 24541 log.cc:824] T 00000000000000000000000000000000 P d8b56e0554174991965bef333e942440: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:42:37 dist-test-slave-mld5 krb5kdc[24460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:37.268757 24541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8b56e0554174991965bef333e942440: No bootstrap required, opened a new log
I20260619 18:42:37.272310 24544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:37.259280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47584 (local address 127.21.143.126:35757)
0619 18:42:37.259699 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:37.259713 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:37.259748 (+ 35us) server_negotiation.cc:407] Connection header received
0619 18:42:37.260532 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:37.260550 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:37.260876 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:37.261212 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:37.262042 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:37.263027 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:37.264018 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:37.264509 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:37.264683 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:37.267340 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:37.267346 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:37.267362 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:37.267389 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:37.269312 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:37.269720 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:37.269726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:37.269731 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:37.269865 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:37.270118 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:37.270122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:37.270124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:37.270598 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:37.270815 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:37.271024 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 18:42:37.271381 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":107,"threads_started":1}
I20260619 18:42:37.273216 24541 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8b56e0554174991965bef333e942440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8b56e0554174991965bef333e942440" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 35757 } }
I20260619 18:42:37.273468 24541 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8b56e0554174991965bef333e942440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:37.273530 24541 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8b56e0554174991965bef333e942440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b56e0554174991965bef333e942440, State: Initialized, Role: FOLLOWER
I20260619 18:42:37.273983 24541 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8b56e0554174991965bef333e942440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8b56e0554174991965bef333e942440" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 35757 } }
I20260619 18:42:37.274102 24541 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8b56e0554174991965bef333e942440 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:42:37.274187 24541 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8b56e0554174991965bef333e942440 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:42:37.274310 24541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8b56e0554174991965bef333e942440 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:37.275241 24541 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8b56e0554174991965bef333e942440 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8b56e0554174991965bef333e942440" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 35757 } }
I20260619 18:42:37.275573 24541 leader_election.cc:302] T 00000000000000000000000000000000 P d8b56e0554174991965bef333e942440 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8b56e0554174991965bef333e942440; no voters:
I20260619 18:42:37.275815 24541 leader_election.cc:288] T 00000000000000000000000000000000 P d8b56e0554174991965bef333e942440 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:42:37.275964 24546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8b56e0554174991965bef333e942440 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:37.276192 24546 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8b56e0554174991965bef333e942440 [term 1 LEADER]: Becoming Leader. State: Replica: d8b56e0554174991965bef333e942440, State: Running, Role: LEADER
I20260619 18:42:37.276484 24546 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8b56e0554174991965bef333e942440 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8b56e0554174991965bef333e942440" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 35757 } }
I20260619 18:42:37.276973 24541 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8b56e0554174991965bef333e942440 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:42:37.278159 24547 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8b56e0554174991965bef333e942440 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8b56e0554174991965bef333e942440" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8b56e0554174991965bef333e942440" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 35757 } } }
I20260619 18:42:37.278303 24547 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8b56e0554174991965bef333e942440 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:37.278160 24548 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8b56e0554174991965bef333e942440 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8b56e0554174991965bef333e942440. Latest consensus state: current_term: 1 leader_uuid: "d8b56e0554174991965bef333e942440" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8b56e0554174991965bef333e942440" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 35757 } } }
I20260619 18:42:37.278395 24548 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8b56e0554174991965bef333e942440 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:37.278715 24555 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:42:37.282214 24555 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:42:37.287707 24555 catalog_manager.cc:1347] Generated new cluster ID: dc2a41953f5c4d93bc05cf2dfb011c52
I20260619 18:42:37.287797 24555 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:42:37.301143 24555 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:42:37.302603 24555 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:42:37.308276 24555 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8b56e0554174991965bef333e942440: Generated new TSK 0
I20260619 18:42:37.308969 24555 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.143.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab.
Entry for principal kudu/127.21.143.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab.
I20260619 18:42:37.338034 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.65:0
--local_ip_for_outbound_sockets=127.21.143.65
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:35757
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:38709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:37.450137 24567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:37.450376 24567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:37.450416 24567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:37.454425 24567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:37.454497 24567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:37.454557 24567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.65
I20260619 18:42:37.459542 24567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:38709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:35757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.65
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:37.460482 24567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:37.461424 24567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:37.470427 24575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:37.470469 24573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:37.470470 24572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:37.470724 24567 server_base.cc:1034] running on GCE node
I20260619 18:42:37.471148 24567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:37.471789 24567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:37.472946 24567 hybrid_clock.cc:648] HybridClock initialized: now 1781894557472935 us; error 25 us; skew 500 ppm
Jun 19 18:42:37 dist-test-slave-mld5 krb5kdc[24460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894557, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:37.475581 24567 init.cc:377] Logged in from keytab as kudu/127.21.143.65@KRBTEST.COM (short username kudu)
I20260619 18:42:37.476419 24567 webserver.cc:460] Webserver started at http://127.21.143.65:43055/ using document root <none> and password file <none>
I20260619 18:42:37.476960 24567 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:37.477049 24567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:37.477347 24567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:37.479439 24567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-0/data/instance:
uuid: "8c7c545c7d4a40b7bfdfc926a11178aa"
format_stamp: "Formatted at 2026-06-19 18:42:37 on dist-test-slave-mld5"
server_key: "4483f927cd23a5c5a1de8252fe39f610"
server_key_iv: "dc0741a7316d3279d212937690a3a064"
server_key_version: "encryptionkey@0"
I20260619 18:42:37.479981 24567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance:
uuid: "8c7c545c7d4a40b7bfdfc926a11178aa"
format_stamp: "Formatted at 2026-06-19 18:42:37 on dist-test-slave-mld5"
server_key: "4483f927cd23a5c5a1de8252fe39f610"
server_key_iv: "dc0741a7316d3279d212937690a3a064"
server_key_version: "encryptionkey@0"
I20260619 18:42:37.483623 24567 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:42:37.486090 24582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:37.487587 24567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:42:37.487717 24567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-0/wal
uuid: "8c7c545c7d4a40b7bfdfc926a11178aa"
format_stamp: "Formatted at 2026-06-19 18:42:37 on dist-test-slave-mld5"
server_key: "4483f927cd23a5c5a1de8252fe39f610"
server_key_iv: "dc0741a7316d3279d212937690a3a064"
server_key_version: "encryptionkey@0"
I20260619 18:42:37.487833 24567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:37.509035 24567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:37.512439 24567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:37.512652 24567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:37.513265 24567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:37.514104 24567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:37.514176 24567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:37.514248 24567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:37.514292 24567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:37.523582 24567 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.65:33299
I20260619 18:42:37.523617 24695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.65:33299 every 8 connection(s)
I20260619 18:42:37.524474 24567 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
Jun 19 18:42:37 dist-test-slave-mld5 krb5kdc[24460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894557, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:37.534361 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 24567
I20260619 18:42:37.534511 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance
I20260619 18:42:37.534857 22077 external_mini_cluster.cc:1428] Setting key 6ea9d30de7098fef8bf4a878d413dc3a
I20260619 18:42:37.538453 24544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:37.526210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:43403 (local address 127.21.143.126:35757)
0619 18:42:37.526370 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:37.526374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:37.527202 (+ 828us) server_negotiation.cc:407] Connection header received
0619 18:42:37.528487 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:37.528489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:37.528536 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:37.528612 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:37.529887 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:37.530427 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:37.531701 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:37.531926 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:37.532006 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:37.534649 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:37.534654 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:37.534659 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:37.534690 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:37.536303 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:37.536835 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:37.536841 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:37.536842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:37.536934 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:37.537283 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:37.537289 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:37.537294 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:37.537565 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:37.537667 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:37.538168 (+ 501us) server_negotiation.cc:299] Negotiation successful
0619 18:42:37.538294 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 18:42:37.539288 24698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:37.526613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:35757 (local address 127.21.143.65:43403)
0619 18:42:37.527046 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:37.527094 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:37.528294 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:37.528842 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:37.528850 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:37.529225 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:37.529741 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:37.529757 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:37.530567 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:37.530570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:37.531585 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:37.531596 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:37.532059 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:37.532062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:37.532205 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:37.532819 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:37.532835 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:37.534380 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:37.536425 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:37.536431 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:37.536449 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:37.536729 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:37.537039 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:37.537041 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:37.537042 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:37.537190 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:37.537665 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:37.537669 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:37.537903 (+ 234us) client_negotiation.cc:769] Sending connection context
0619 18:42:37.538126 (+ 223us) client_negotiation.cc:240] Negotiation successful
0619 18:42:37.538455 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":98,"threads_started":1}
I20260619 18:42:37.540675 24696 heartbeater.cc:347] Connected to a master server at 127.21.143.126:35757
I20260619 18:42:37.540915 24696 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:37.541394 24696 heartbeater.cc:511] Master 127.21.143.126:35757 requested a full tablet report, sending...
I20260619 18:42:37.543084 24505 ts_manager.cc:194] Registered new tserver with Master: 8c7c545c7d4a40b7bfdfc926a11178aa (127.21.143.65:33299)
I20260619 18:42:37.544801 24505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.65@KRBTEST.COM'} at 127.21.143.65:43403
WARNING: no policy specified for kudu/127.21.143.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab.
Entry for principal kudu/127.21.143.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab.
I20260619 18:42:37.561735 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.66:0
--local_ip_for_outbound_sockets=127.21.143.66
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:35757
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:38709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:37.671461 24701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:37.671715 24701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:37.671789 24701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:37.675762 24701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:37.675810 24701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:37.675910 24701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.66
I20260619 18:42:37.681245 24701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:38709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:35757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.66
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:37.682210 24701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:37.683162 24701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:37.692324 24707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:37.692389 24706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:37.692337 24709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:37.692526 24701 server_base.cc:1034] running on GCE node
I20260619 18:42:37.693006 24701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:37.693619 24701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:37.694800 24701 hybrid_clock.cc:648] HybridClock initialized: now 1781894557694768 us; error 42 us; skew 500 ppm
Jun 19 18:42:37 dist-test-slave-mld5 krb5kdc[24460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894557, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:37.697486 24701 init.cc:377] Logged in from keytab as kudu/127.21.143.66@KRBTEST.COM (short username kudu)
I20260619 18:42:37.698304 24701 webserver.cc:460] Webserver started at http://127.21.143.66:37269/ using document root <none> and password file <none>
I20260619 18:42:37.698856 24701 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:37.698947 24701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:37.699174 24701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:37.701180 24701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-1/data/instance:
uuid: "676ca194414f45408db9e72e738b15d0"
format_stamp: "Formatted at 2026-06-19 18:42:37 on dist-test-slave-mld5"
server_key: "80adbb99aa2719ef3eb2b1a8c72e81fa"
server_key_iv: "b2ffd2e29cf5b2d847597cba69269874"
server_key_version: "encryptionkey@0"
I20260619 18:42:37.701721 24701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance:
uuid: "676ca194414f45408db9e72e738b15d0"
format_stamp: "Formatted at 2026-06-19 18:42:37 on dist-test-slave-mld5"
server_key: "80adbb99aa2719ef3eb2b1a8c72e81fa"
server_key_iv: "b2ffd2e29cf5b2d847597cba69269874"
server_key_version: "encryptionkey@0"
I20260619 18:42:37.705304 24701 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:42:37.707660 24716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:37.708789 24701 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:42:37.708925 24701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-1/wal
uuid: "676ca194414f45408db9e72e738b15d0"
format_stamp: "Formatted at 2026-06-19 18:42:37 on dist-test-slave-mld5"
server_key: "80adbb99aa2719ef3eb2b1a8c72e81fa"
server_key_iv: "b2ffd2e29cf5b2d847597cba69269874"
server_key_version: "encryptionkey@0"
I20260619 18:42:37.709035 24701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:37.724105 24701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:37.726833 24701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:37.727031 24701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:37.727622 24701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:37.728422 24701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:37.728490 24701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:37.728556 24701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:37.728606 24701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:37.737578 24701 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.66:44125
I20260619 18:42:37.737607 24829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.66:44125 every 8 connection(s)
I20260619 18:42:37.738413 24701 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
Jun 19 18:42:37 dist-test-slave-mld5 krb5kdc[24460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894557, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:37.747664 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 24701
I20260619 18:42:37.747787 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance
I20260619 18:42:37.748080 22077 external_mini_cluster.cc:1428] Setting key aa8791b3800d33c514989b82ed04abd0
I20260619 18:42:37.751559 24544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:37.740102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:48837 (local address 127.21.143.126:35757)
0619 18:42:37.740225 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:37.740229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:37.741028 (+ 799us) server_negotiation.cc:407] Connection header received
0619 18:42:37.742277 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:37.742280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:37.742335 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:37.742434 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:37.743565 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:37.744068 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:37.745212 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:37.745440 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:37.745562 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:37.748016 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:37.748020 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:37.748024 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:37.748052 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:37.749520 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:37.750039 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:37.750044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:37.750046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:37.750133 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:37.750468 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:37.750473 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:37.750474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:37.750658 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:37.750761 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:37.751248 (+ 487us) server_negotiation.cc:299] Negotiation successful
0619 18:42:37.751407 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 18:42:37.752401 24832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:37.740497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:35757 (local address 127.21.143.66:48837)
0619 18:42:37.740883 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:37.740921 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:37.742071 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:37.742606 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:37.742612 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:37.742934 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:37.743424 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:37.743438 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:37.744192 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:37.744197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:37.745091 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:37.745101 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:37.745618 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:37.745622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:37.745781 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:37.746343 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:37.746358 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:37.747846 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:37.749640 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:37.749646 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:37.749660 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:37.749922 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:37.750230 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:37.750233 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:37.750235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:37.750375 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:37.750769 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:37.750777 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:37.751019 (+ 242us) client_negotiation.cc:769] Sending connection context
0619 18:42:37.751206 (+ 187us) client_negotiation.cc:240] Negotiation successful
0619 18:42:37.751570 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":93,"threads_started":1}
I20260619 18:42:37.753547 24830 heartbeater.cc:347] Connected to a master server at 127.21.143.126:35757
I20260619 18:42:37.753789 24830 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:37.754251 24830 heartbeater.cc:511] Master 127.21.143.126:35757 requested a full tablet report, sending...
I20260619 18:42:37.755287 24505 ts_manager.cc:194] Registered new tserver with Master: 676ca194414f45408db9e72e738b15d0 (127.21.143.66:44125)
I20260619 18:42:37.756137 24505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.66@KRBTEST.COM'} at 127.21.143.66:48837
WARNING: no policy specified for kudu/127.21.143.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab.
Entry for principal kudu/127.21.143.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab.
I20260619 18:42:37.772470 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.67:0
--local_ip_for_outbound_sockets=127.21.143.67
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:35757
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:38709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:37.879899 24835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:37.880291 24835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:37.880364 24835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:37.884517 24835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:37.884613 24835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:37.884706 24835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.67
I20260619 18:42:37.890074 24835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:38709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:35757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.67
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:37.891139 24835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:37.892233 24835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:37.901453 24843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:37.901466 24841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:37.901551 24835 server_base.cc:1034] running on GCE node
W20260619 18:42:37.901474 24840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:37.902130 24835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:37.902663 24835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:37.903846 24835 hybrid_clock.cc:648] HybridClock initialized: now 1781894557903833 us; error 24 us; skew 500 ppm
Jun 19 18:42:37 dist-test-slave-mld5 krb5kdc[24460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894557, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:37.906394 24835 init.cc:377] Logged in from keytab as kudu/127.21.143.67@KRBTEST.COM (short username kudu)
I20260619 18:42:37.907286 24835 webserver.cc:460] Webserver started at http://127.21.143.67:32793/ using document root <none> and password file <none>
I20260619 18:42:37.907861 24835 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:37.907948 24835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:37.908143 24835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:37.910130 24835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-2/data/instance:
uuid: "2af50637b0c84a8cb2d703ca8d655952"
format_stamp: "Formatted at 2026-06-19 18:42:37 on dist-test-slave-mld5"
server_key: "41ca33ece8cd347373b5a4237ba7d41a"
server_key_iv: "ae9e26e5f1991c9d8403edf3b8e3ce82"
server_key_version: "encryptionkey@0"
I20260619 18:42:37.910688 24835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance:
uuid: "2af50637b0c84a8cb2d703ca8d655952"
format_stamp: "Formatted at 2026-06-19 18:42:37 on dist-test-slave-mld5"
server_key: "41ca33ece8cd347373b5a4237ba7d41a"
server_key_iv: "ae9e26e5f1991c9d8403edf3b8e3ce82"
server_key_version: "encryptionkey@0"
I20260619 18:42:37.914242 24835 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260619 18:42:37.916535 24850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:37.917728 24835 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:42:37.917866 24835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-2/wal
uuid: "2af50637b0c84a8cb2d703ca8d655952"
format_stamp: "Formatted at 2026-06-19 18:42:37 on dist-test-slave-mld5"
server_key: "41ca33ece8cd347373b5a4237ba7d41a"
server_key_iv: "ae9e26e5f1991c9d8403edf3b8e3ce82"
server_key_version: "encryptionkey@0"
I20260619 18:42:37.917966 24835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:37.934792 24835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:37.937711 24835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:37.937891 24835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:37.938426 24835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:37.939239 24835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:37.939289 24835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:37.939424 24835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:37.939459 24835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:37.948382 24835 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.67:40409
I20260619 18:42:37.948398 24963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.67:40409 every 8 connection(s)
I20260619 18:42:37.949303 24835 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
I20260619 18:42:37.958055 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 24835
I20260619 18:42:37.958148 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance
Jun 19 18:42:37 dist-test-slave-mld5 krb5kdc[24460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894557, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:37.958472 22077 external_mini_cluster.cc:1428] Setting key 6be019c6c2e71e59599f8e09518dfe30
I20260619 18:42:37.962353 24544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:37.951055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:56899 (local address 127.21.143.126:35757)
0619 18:42:37.951223 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:37.951228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:37.952174 (+ 946us) server_negotiation.cc:407] Connection header received
0619 18:42:37.953360 (+ 1186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:37.953362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:37.953410 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:37.953483 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:37.954587 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:37.955103 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:37.956160 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:37.956360 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:37.956435 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:37.958902 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:37.958906 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:37.958910 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:37.958943 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:37.960657 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:37.961144 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:37.961146 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:37.961147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:37.961218 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:37.961483 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:37.961485 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:37.961487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:37.961657 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:37.961747 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:37.962153 (+ 406us) server_negotiation.cc:299] Negotiation successful
0619 18:42:37.962241 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 18:42:37.963183 24966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:37.951511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:35757 (local address 127.21.143.67:56899)
0619 18:42:37.952028 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:37.952069 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:37.953174 (+ 1105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:37.953651 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:37.953658 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:37.953994 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:37.954465 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:37.954476 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:37.955203 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:37.955205 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:37.956064 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:37.956072 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:37.956479 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:37.956482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:37.956609 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:37.957251 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:37.957267 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:37.958748 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:37.960791 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:37.960796 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:37.960811 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:37.961063 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:37.961291 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:37.961293 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:37.961295 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:37.961411 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:37.961752 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:37.961757 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:37.961962 (+ 205us) client_negotiation.cc:769] Sending connection context
0619 18:42:37.962109 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 18:42:37.962341 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":133,"threads_started":1}
I20260619 18:42:37.964203 24964 heartbeater.cc:347] Connected to a master server at 127.21.143.126:35757
I20260619 18:42:37.964427 24964 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:37.964874 24964 heartbeater.cc:511] Master 127.21.143.126:35757 requested a full tablet report, sending...
I20260619 18:42:37.965794 24505 ts_manager.cc:194] Registered new tserver with Master: 2af50637b0c84a8cb2d703ca8d655952 (127.21.143.67:40409)
I20260619 18:42:37.966339 24505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.67@KRBTEST.COM'} at 127.21.143.67:56899
I20260619 18:42:37.972275 22077 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 18:42:37.984560 24544 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:37.983153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47596 (local address 127.21.143.126:35757)
0619 18:42:37.983273 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:37.983276 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:37.983495 (+ 219us) server_negotiation.cc:407] Connection header received
0619 18:42:37.983645 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:37.983648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:37.983688 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:37.983759 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:37.984409 (+ 650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47596: BlockingRecv error: recv got EOF from 127.0.0.1:47596 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 18:42:37.985767 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 24567
I20260619 18:42:37.991113 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 24701
I20260619 18:42:37.995937 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 24835
I20260619 18:42:38.000901 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 24473
2026-06-19T18:42:37Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2067 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:42:38 dist-test-slave-mld5 krb5kdc[24976](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:42:38 dist-test-slave-mld5 krb5kdc[24976](info): set up 2 sockets
Jun 19 18:42:38 dist-test-slave-mld5 krb5kdc[24976](info): commencing operation
krb5kdc: starting...
W20260619 18:42:39.027364 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:42:39 dist-test-slave-mld5 krb5kdc[24976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:42:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:42:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.143.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
I20260619 18:42:39.140336 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:38213
--webserver_interface=127.21.143.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:38213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:39.248273 24989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:39.248540 24989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:39.248615 24989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:39.252673 24989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:42:39.252738 24989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:39.252758 24989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:39.252820 24989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:42:39.252854 24989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:42:39.258311 24989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:38213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:38213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:39.259470 24989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:39.260505 24989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:39.268517 24994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:39.268612 24995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:39.268513 24997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:39.268772 24989 server_base.cc:1034] running on GCE node
I20260619 18:42:39.269439 24989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:39.270442 24989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:39.271588 24989 hybrid_clock.cc:648] HybridClock initialized: now 1781894559271554 us; error 47 us; skew 500 ppm
Jun 19 18:42:39 dist-test-slave-mld5 krb5kdc[24976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894559, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:39.274219 24989 init.cc:377] Logged in from keytab as kudu/127.21.143.126@KRBTEST.COM (short username kudu)
I20260619 18:42:39.274982 24989 webserver.cc:460] Webserver started at http://127.21.143.126:35819/ using document root <none> and password file <none>
I20260619 18:42:39.275545 24989 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:39.275606 24989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:39.275769 24989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:39.277707 24989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "a07c88af53544710ace47cc2ef6d7878"
format_stamp: "Formatted at 2026-06-19 18:42:39 on dist-test-slave-mld5"
server_key: "a15c3852525be14e0d7932dec9b2258e"
server_key_iv: "256f5f6dab51874da7bafa86c0ecb3cd"
server_key_version: "encryptionkey@0"
I20260619 18:42:39.278219 24989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "a07c88af53544710ace47cc2ef6d7878"
format_stamp: "Formatted at 2026-06-19 18:42:39 on dist-test-slave-mld5"
server_key: "a15c3852525be14e0d7932dec9b2258e"
server_key_iv: "256f5f6dab51874da7bafa86c0ecb3cd"
server_key_version: "encryptionkey@0"
I20260619 18:42:39.281848 24989 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 18:42:39.284180 25004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:39.285426 24989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:42:39.285566 24989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "a07c88af53544710ace47cc2ef6d7878"
format_stamp: "Formatted at 2026-06-19 18:42:39 on dist-test-slave-mld5"
server_key: "a15c3852525be14e0d7932dec9b2258e"
server_key_iv: "256f5f6dab51874da7bafa86c0ecb3cd"
server_key_version: "encryptionkey@0"
I20260619 18:42:39.285688 24989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:39.304379 24989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:39.307332 24989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:39.307541 24989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:39.314299 24989 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:38213
I20260619 18:42:39.314580 25056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:38213 every 8 connection(s)
I20260619 18:42:39.315282 24989 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:42:39.316490 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 24989
I20260619 18:42:39.316614 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:42:39.316924 22077 external_mini_cluster.cc:1428] Setting key 8b7612787871cb64275318f4e3980fa4
I20260619 18:42:39.318683 25057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:39.325654 25057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a07c88af53544710ace47cc2ef6d7878: Bootstrap starting.
Jun 19 18:42:39 dist-test-slave-mld5 krb5kdc[24976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:39.328583 25057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a07c88af53544710ace47cc2ef6d7878: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:39.329483 25057 log.cc:824] T 00000000000000000000000000000000 P a07c88af53544710ace47cc2ef6d7878: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:39.331476 25057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a07c88af53544710ace47cc2ef6d7878: No bootstrap required, opened a new log
I20260619 18:42:39.332186 25060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:39.318201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53866 (local address 127.21.143.126:38213)
0619 18:42:39.318620 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:39.318639 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:39.318681 (+ 42us) server_negotiation.cc:407] Connection header received
0619 18:42:39.319556 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:39.319580 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:39.320002 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:39.320390 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:39.321088 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:39.322634 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:39.323539 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:39.324230 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:39.324405 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:39.327064 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:39.327071 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:39.327087 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:39.327120 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:39.329203 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:39.329700 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:39.329705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:39.329710 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:39.329823 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:39.330117 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:39.330127 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:39.330128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:39.330539 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:39.330738 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:39.331049 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 18:42:39.331410 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":116,"threads_started":1}
I20260619 18:42:39.335045 22077 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:42:39.335223 25057 raft_consensus.cc:348] T 00000000000000000000000000000000 P a07c88af53544710ace47cc2ef6d7878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07c88af53544710ace47cc2ef6d7878" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 38213 } }
I20260619 18:42:39.335493 25057 raft_consensus.cc:374] T 00000000000000000000000000000000 P a07c88af53544710ace47cc2ef6d7878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:39.335567 25057 raft_consensus.cc:729] T 00000000000000000000000000000000 P a07c88af53544710ace47cc2ef6d7878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a07c88af53544710ace47cc2ef6d7878, State: Initialized, Role: FOLLOWER
I20260619 18:42:39.335963 25057 consensus_queue.cc:260] T 00000000000000000000000000000000 P a07c88af53544710ace47cc2ef6d7878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07c88af53544710ace47cc2ef6d7878" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 38213 } }
I20260619 18:42:39.336073 25057 raft_consensus.cc:388] T 00000000000000000000000000000000 P a07c88af53544710ace47cc2ef6d7878 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:42:39.336153 25057 raft_consensus.cc:482] T 00000000000000000000000000000000 P a07c88af53544710ace47cc2ef6d7878 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:42:39.336242 25057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a07c88af53544710ace47cc2ef6d7878 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:39.337333 25057 raft_consensus.cc:504] T 00000000000000000000000000000000 P a07c88af53544710ace47cc2ef6d7878 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07c88af53544710ace47cc2ef6d7878" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 38213 } }
I20260619 18:42:39.337710 25057 leader_election.cc:302] T 00000000000000000000000000000000 P a07c88af53544710ace47cc2ef6d7878 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a07c88af53544710ace47cc2ef6d7878; no voters:
I20260619 18:42:39.338083 25057 leader_election.cc:288] T 00000000000000000000000000000000 P a07c88af53544710ace47cc2ef6d7878 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:42:39.338169 25063 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a07c88af53544710ace47cc2ef6d7878 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:39.338415 25063 raft_consensus.cc:686] T 00000000000000000000000000000000 P a07c88af53544710ace47cc2ef6d7878 [term 1 LEADER]: Becoming Leader. State: Replica: a07c88af53544710ace47cc2ef6d7878, State: Running, Role: LEADER
I20260619 18:42:39.338706 25063 consensus_queue.cc:237] T 00000000000000000000000000000000 P a07c88af53544710ace47cc2ef6d7878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07c88af53544710ace47cc2ef6d7878" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 38213 } }
I20260619 18:42:39.339455 25057 sys_catalog.cc:564] T 00000000000000000000000000000000 P a07c88af53544710ace47cc2ef6d7878 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:42:39.340282 25064 sys_catalog.cc:455] T 00000000000000000000000000000000 P a07c88af53544710ace47cc2ef6d7878 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a07c88af53544710ace47cc2ef6d7878" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07c88af53544710ace47cc2ef6d7878" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 38213 } } }
I20260619 18:42:39.340406 25064 sys_catalog.cc:458] T 00000000000000000000000000000000 P a07c88af53544710ace47cc2ef6d7878 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:39.340780 25065 sys_catalog.cc:455] T 00000000000000000000000000000000 P a07c88af53544710ace47cc2ef6d7878 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a07c88af53544710ace47cc2ef6d7878. Latest consensus state: current_term: 1 leader_uuid: "a07c88af53544710ace47cc2ef6d7878" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07c88af53544710ace47cc2ef6d7878" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 38213 } } }
I20260619 18:42:39.340925 25065 sys_catalog.cc:458] T 00000000000000000000000000000000 P a07c88af53544710ace47cc2ef6d7878 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:39.341717 25069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:42:39.344745 25069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:42:39.350627 25069 catalog_manager.cc:1347] Generated new cluster ID: 8ff5bb9d8c704b8882345e61dd7e2fe8
I20260619 18:42:39.350687 25069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:42:39.353430 25060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:39.347989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53874 (local address 127.21.143.126:38213)
0619 18:42:39.348098 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:39.348102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:39.348368 (+ 266us) server_negotiation.cc:407] Connection header received
0619 18:42:39.348468 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:39.348472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:39.348523 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:39.348606 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:39.349225 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:39.350000 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:39.351160 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:39.351505 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:39.351598 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:39.352515 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:39.352518 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:39.352521 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:42:39.352999 (+ 478us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:39.353029 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:42:39.353046 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:39.353135 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:39.353261 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 18:42:39.353315 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 18:42:39.377838 25069 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:42:39.378767 25069 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:42:39.393066 25069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a07c88af53544710ace47cc2ef6d7878: Generated new TSK 0
I20260619 18:42:39.393752 25069 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:42:39.399919 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 24989
2026-06-19T18:42:39Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1398 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:42:39 dist-test-slave-mld5 krb5kdc[25090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:42:39 dist-test-slave-mld5 krb5kdc[25090](info): set up 2 sockets
Jun 19 18:42:39 dist-test-slave-mld5 krb5kdc[25090](info): commencing operation
krb5kdc: starting...
W20260619 18:42:40.435408 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:42:40 dist-test-slave-mld5 krb5kdc[25090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:42:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:42:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.143.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
I20260619 18:42:40.547909 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:42293
--webserver_interface=127.21.143.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:42691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:42293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:40.656092 25103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:40.656394 25103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:40.656481 25103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:40.660557 25103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:42:40.660636 25103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:40.660673 25103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:40.660712 25103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:42:40.660746 25103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:42:40.665868 25103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:42691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:42293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:42293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:40.666945 25103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:40.667956 25103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:40.676335 25109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:40.676342 25111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:40.676342 25108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:40.676456 25103 server_base.cc:1034] running on GCE node
I20260619 18:42:40.677196 25103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:40.678297 25103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:40.679461 25103 hybrid_clock.cc:648] HybridClock initialized: now 1781894560679448 us; error 24 us; skew 500 ppm
Jun 19 18:42:40 dist-test-slave-mld5 krb5kdc[25090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894560, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:40.682085 25103 init.cc:377] Logged in from keytab as kudu/127.21.143.126@KRBTEST.COM (short username kudu)
I20260619 18:42:40.682996 25103 webserver.cc:460] Webserver started at http://127.21.143.126:40337/ using document root <none> and password file <none>
I20260619 18:42:40.683615 25103 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:40.683674 25103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:40.683917 25103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:40.685930 25103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "ed2399d1cfd64d4494c8d203631a93f0"
format_stamp: "Formatted at 2026-06-19 18:42:40 on dist-test-slave-mld5"
server_key: "63b368167d04e38eaa9b27f424cb4a21"
server_key_iv: "1a9d8a11616b7dd7c878720d8a1e4fbd"
server_key_version: "encryptionkey@0"
I20260619 18:42:40.686482 25103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "ed2399d1cfd64d4494c8d203631a93f0"
format_stamp: "Formatted at 2026-06-19 18:42:40 on dist-test-slave-mld5"
server_key: "63b368167d04e38eaa9b27f424cb4a21"
server_key_iv: "1a9d8a11616b7dd7c878720d8a1e4fbd"
server_key_version: "encryptionkey@0"
I20260619 18:42:40.690145 25103 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 18:42:40.692528 25118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:40.693743 25103 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:42:40.693838 25103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "ed2399d1cfd64d4494c8d203631a93f0"
format_stamp: "Formatted at 2026-06-19 18:42:40 on dist-test-slave-mld5"
server_key: "63b368167d04e38eaa9b27f424cb4a21"
server_key_iv: "1a9d8a11616b7dd7c878720d8a1e4fbd"
server_key_version: "encryptionkey@0"
I20260619 18:42:40.693964 25103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:40.711194 25103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:40.713971 25103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:40.714170 25103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:40.721081 25103 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:42293
I20260619 18:42:40.721403 25170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:42293 every 8 connection(s)
I20260619 18:42:40.722115 25103 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:42:40.723408 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 25103
I20260619 18:42:40.723546 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:42:40.723824 22077 external_mini_cluster.cc:1428] Setting key 4999423c572ec9a480b10dde0ee1600b
I20260619 18:42:40.725178 25171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:40.731887 25171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed2399d1cfd64d4494c8d203631a93f0: Bootstrap starting.
Jun 19 18:42:40 dist-test-slave-mld5 krb5kdc[25090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:40.734300 25171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed2399d1cfd64d4494c8d203631a93f0: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:40.735212 25171 log.cc:824] T 00000000000000000000000000000000 P ed2399d1cfd64d4494c8d203631a93f0: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:40.737092 25171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed2399d1cfd64d4494c8d203631a93f0: No bootstrap required, opened a new log
I20260619 18:42:40.738579 25174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:40.725171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37112 (local address 127.21.143.126:42293)
0619 18:42:40.725610 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:40.725628 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:40.725669 (+ 41us) server_negotiation.cc:407] Connection header received
0619 18:42:40.726493 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:40.726515 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:40.726907 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:40.727267 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:40.727979 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:40.729467 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:40.730310 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:40.731021 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:40.731184 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:40.733625 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:40.733631 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:40.733647 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:40.733677 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:40.735659 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:40.736109 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:40.736115 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:40.736123 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:40.736234 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:40.736516 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:40.736521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:40.736523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:40.736901 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:40.737105 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:40.737395 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 18:42:40.737688 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":119,"threads_started":1}
I20260619 18:42:40.741819 25171 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed2399d1cfd64d4494c8d203631a93f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2399d1cfd64d4494c8d203631a93f0" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42293 } }
I20260619 18:42:40.742022 25171 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed2399d1cfd64d4494c8d203631a93f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:40.742105 25171 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed2399d1cfd64d4494c8d203631a93f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed2399d1cfd64d4494c8d203631a93f0, State: Initialized, Role: FOLLOWER
I20260619 18:42:40.742509 25171 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed2399d1cfd64d4494c8d203631a93f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2399d1cfd64d4494c8d203631a93f0" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42293 } }
I20260619 18:42:40.742645 25171 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed2399d1cfd64d4494c8d203631a93f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:42:40.742738 25171 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed2399d1cfd64d4494c8d203631a93f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:42:40.742843 25171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed2399d1cfd64d4494c8d203631a93f0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:40.743820 25171 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed2399d1cfd64d4494c8d203631a93f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2399d1cfd64d4494c8d203631a93f0" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42293 } }
I20260619 18:42:40.744169 25171 leader_election.cc:302] T 00000000000000000000000000000000 P ed2399d1cfd64d4494c8d203631a93f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed2399d1cfd64d4494c8d203631a93f0; no voters:
I20260619 18:42:40.744499 25171 leader_election.cc:288] T 00000000000000000000000000000000 P ed2399d1cfd64d4494c8d203631a93f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:42:40.744609 25176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed2399d1cfd64d4494c8d203631a93f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:40.744832 25176 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed2399d1cfd64d4494c8d203631a93f0 [term 1 LEADER]: Becoming Leader. State: Replica: ed2399d1cfd64d4494c8d203631a93f0, State: Running, Role: LEADER
I20260619 18:42:40.745205 25176 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed2399d1cfd64d4494c8d203631a93f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2399d1cfd64d4494c8d203631a93f0" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42293 } }
I20260619 18:42:40.745733 25171 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed2399d1cfd64d4494c8d203631a93f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:42:40.746825 25178 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed2399d1cfd64d4494c8d203631a93f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed2399d1cfd64d4494c8d203631a93f0. Latest consensus state: current_term: 1 leader_uuid: "ed2399d1cfd64d4494c8d203631a93f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2399d1cfd64d4494c8d203631a93f0" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42293 } } }
I20260619 18:42:40.746977 25178 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed2399d1cfd64d4494c8d203631a93f0 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:40.747340 25177 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed2399d1cfd64d4494c8d203631a93f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed2399d1cfd64d4494c8d203631a93f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2399d1cfd64d4494c8d203631a93f0" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42293 } } }
I20260619 18:42:40.747463 25185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:42:40.750830 25185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:42:40.751400 25177 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed2399d1cfd64d4494c8d203631a93f0 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:40.755978 25185 catalog_manager.cc:1347] Generated new cluster ID: 2356dd4eee8146c697af984e80c90ac3
I20260619 18:42:40.756036 25185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:42:40.772120 25185 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:42:40.773252 25185 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:42:40.778865 25185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed2399d1cfd64d4494c8d203631a93f0: Generated new TSK 0
I20260619 18:42:40.779577 25185 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.143.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab.
Entry for principal kudu/127.21.143.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab.
I20260619 18:42:40.810894 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.65:0
--local_ip_for_outbound_sockets=127.21.143.65
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:42293
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:42691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:40.919936 25197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:40.920179 25197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:40.920249 25197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:40.924268 25197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:40.924315 25197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:40.924388 25197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.65
I20260619 18:42:40.929476 25197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:42691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:42293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.65
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:40.930508 25197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:40.931573 25197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:40.940682 25205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:40.940689 25202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:40.940689 25203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:40.940721 25197 server_base.cc:1034] running on GCE node
I20260619 18:42:40.941265 25197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:40.941871 25197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:40.943027 25197 hybrid_clock.cc:648] HybridClock initialized: now 1781894560943010 us; error 27 us; skew 500 ppm
Jun 19 18:42:40 dist-test-slave-mld5 krb5kdc[25090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894560, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:40.945701 25197 init.cc:377] Logged in from keytab as kudu/127.21.143.65@KRBTEST.COM (short username kudu)
I20260619 18:42:40.946691 25197 webserver.cc:460] Webserver started at http://127.21.143.65:40613/ using document root <none> and password file <none>
I20260619 18:42:40.947430 25197 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:40.947497 25197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:40.947711 25197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:40.949615 25197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-0/data/instance:
uuid: "edc9d0f4ebe94b88a1964b92bccd0d3d"
format_stamp: "Formatted at 2026-06-19 18:42:40 on dist-test-slave-mld5"
server_key: "3ae63e38139ad2a383e67cb02fc482c3"
server_key_iv: "35afe0924c071215c8efe839b6d96c1c"
server_key_version: "encryptionkey@0"
I20260619 18:42:40.950155 25197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance:
uuid: "edc9d0f4ebe94b88a1964b92bccd0d3d"
format_stamp: "Formatted at 2026-06-19 18:42:40 on dist-test-slave-mld5"
server_key: "3ae63e38139ad2a383e67cb02fc482c3"
server_key_iv: "35afe0924c071215c8efe839b6d96c1c"
server_key_version: "encryptionkey@0"
I20260619 18:42:40.953747 25197 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:42:40.956024 25212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:40.957151 25197 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 18:42:40.957320 25197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-0/wal
uuid: "edc9d0f4ebe94b88a1964b92bccd0d3d"
format_stamp: "Formatted at 2026-06-19 18:42:40 on dist-test-slave-mld5"
server_key: "3ae63e38139ad2a383e67cb02fc482c3"
server_key_iv: "35afe0924c071215c8efe839b6d96c1c"
server_key_version: "encryptionkey@0"
I20260619 18:42:40.957424 25197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:40.976197 25197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:40.979010 25197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:40.979219 25197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:40.979851 25197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:40.980633 25197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:40.980700 25197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:40.980772 25197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:40.980825 25197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:40.990079 25197 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.65:34445
I20260619 18:42:40.990099 25325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.65:34445 every 8 connection(s)
I20260619 18:42:40.990974 25197 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
I20260619 18:42:40.997026 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 25197
I20260619 18:42:40.997154 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance
I20260619 18:42:40.997447 22077 external_mini_cluster.cc:1428] Setting key 10cc141239b0f889a9cc569a05eea8e9
Jun 19 18:42:40 dist-test-slave-mld5 krb5kdc[25090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894560, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:41.005344 25174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:40.992796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:48363 (local address 127.21.143.126:42293)
0619 18:42:40.992957 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:40.992962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:40.993822 (+ 860us) server_negotiation.cc:407] Connection header received
0619 18:42:40.995110 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:40.995113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:40.995163 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:40.995235 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:40.996476 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:40.997121 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:40.998500 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:40.998822 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:40.998935 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:41.001381 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:41.001385 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:41.001388 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:41.001417 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:41.003047 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:41.003659 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:41.003665 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:41.003668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:41.003765 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:41.004109 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:41.004116 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:41.004119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:41.004394 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:41.004530 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:41.004994 (+ 464us) server_negotiation.cc:299] Negotiation successful
0619 18:42:41.005147 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 18:42:41.006090 25328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:40.993230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42293 (local address 127.21.143.65:48363)
0619 18:42:40.993674 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:40.993717 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:40.994908 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:40.995458 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:40.995465 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:40.995817 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:40.996325 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:40.996339 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:40.997256 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:40.997260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:40.998369 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:40.998379 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:40.998965 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:40.998969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:40.999146 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:40.999796 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:40.999813 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:41.001215 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:41.003183 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:41.003195 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:41.003215 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:41.003559 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:41.003863 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:41.003865 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:41.003866 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:41.003998 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:41.004511 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:41.004516 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:41.004744 (+ 228us) client_negotiation.cc:769] Sending connection context
0619 18:42:41.004890 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 18:42:41.005138 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":108,"threads_started":1}
I20260619 18:42:41.007396 25326 heartbeater.cc:347] Connected to a master server at 127.21.143.126:42293
WARNING: no policy specified for kudu/127.21.143.66@KRBTEST.COM; defaulting to no policy
I20260619 18:42:41.007666 25326 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:41.008205 25326 heartbeater.cc:511] Master 127.21.143.126:42293 requested a full tablet report, sending...
I20260619 18:42:41.009759 25135 ts_manager.cc:194] Registered new tserver with Master: edc9d0f4ebe94b88a1964b92bccd0d3d (127.21.143.65:34445)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.66@KRBTEST.COM" created.
I20260619 18:42:41.010978 25135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.65@KRBTEST.COM'} at 127.21.143.65:48363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab.
Entry for principal kudu/127.21.143.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab.
I20260619 18:42:41.024816 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.66:0
--local_ip_for_outbound_sockets=127.21.143.66
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:42293
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:42691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:41.137436 25331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:41.137682 25331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:41.137723 25331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:41.141907 25331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:41.141961 25331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:41.142051 25331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.66
I20260619 18:42:41.147462 25331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:42691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:42293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.66
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:41.148530 25331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:41.149580 25331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:41.159370 25339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:41.159395 25336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:41.159370 25337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:41.159600 25331 server_base.cc:1034] running on GCE node
I20260619 18:42:41.160153 25331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:41.160776 25331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:41.161931 25331 hybrid_clock.cc:648] HybridClock initialized: now 1781894561161876 us; error 65 us; skew 500 ppm
Jun 19 18:42:41 dist-test-slave-mld5 krb5kdc[25090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894561, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:41.164659 25331 init.cc:377] Logged in from keytab as kudu/127.21.143.66@KRBTEST.COM (short username kudu)
I20260619 18:42:41.165468 25331 webserver.cc:460] Webserver started at http://127.21.143.66:35867/ using document root <none> and password file <none>
I20260619 18:42:41.166085 25331 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:41.166177 25331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:41.166393 25331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:41.168529 25331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-1/data/instance:
uuid: "4c123c835f7a43c69aa6f083c984536c"
format_stamp: "Formatted at 2026-06-19 18:42:41 on dist-test-slave-mld5"
server_key: "a5fac81b5365675c977abccebbede703"
server_key_iv: "8ac2f98b7089a4a72344f5ee0cb0c5ef"
server_key_version: "encryptionkey@0"
I20260619 18:42:41.169263 25331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance:
uuid: "4c123c835f7a43c69aa6f083c984536c"
format_stamp: "Formatted at 2026-06-19 18:42:41 on dist-test-slave-mld5"
server_key: "a5fac81b5365675c977abccebbede703"
server_key_iv: "8ac2f98b7089a4a72344f5ee0cb0c5ef"
server_key_version: "encryptionkey@0"
I20260619 18:42:41.172824 25331 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:42:41.175052 25346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:41.176281 25331 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:42:41.176370 25331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-1/wal
uuid: "4c123c835f7a43c69aa6f083c984536c"
format_stamp: "Formatted at 2026-06-19 18:42:41 on dist-test-slave-mld5"
server_key: "a5fac81b5365675c977abccebbede703"
server_key_iv: "8ac2f98b7089a4a72344f5ee0cb0c5ef"
server_key_version: "encryptionkey@0"
I20260619 18:42:41.176447 25331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:41.198256 25331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:41.201841 25331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:41.202001 25331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:41.202565 25331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:41.203485 25331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:41.203540 25331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:41.203585 25331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:41.203600 25331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:41.212296 25331 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.66:42707
I20260619 18:42:41.212399 25459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.66:42707 every 8 connection(s)
I20260619 18:42:41.213227 25331 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
I20260619 18:42:41.221437 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 25331
I20260619 18:42:41.221593 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance
I20260619 18:42:41.221930 22077 external_mini_cluster.cc:1428] Setting key 8fd0e231794f4d76bd5096e491c7cd29
Jun 19 18:42:41 dist-test-slave-mld5 krb5kdc[25090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894561, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:41.231413 25174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:41.215258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:41987 (local address 127.21.143.126:42293)
0619 18:42:41.215487 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:41.215492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:41.216633 (+ 1141us) server_negotiation.cc:407] Connection header received
0619 18:42:41.218784 (+ 2151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:41.218788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:41.218866 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:41.218980 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:41.220797 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:41.221854 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:41.223703 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:41.224037 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:41.224162 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:41.227183 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:41.227186 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:41.227188 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:41.227214 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:41.228880 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:41.229585 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:41.229588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:41.229589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:41.229665 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:41.230090 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:41.230093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:41.230094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:41.230294 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:41.230371 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:41.231098 (+ 727us) server_negotiation.cc:299] Negotiation successful
0619 18:42:41.231209 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
WARNING: no policy specified for kudu/127.21.143.67@KRBTEST.COM; defaulting to no policy
I20260619 18:42:41.232642 25462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:41.215834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42293 (local address 127.21.143.66:41987)
0619 18:42:41.216437 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:41.216493 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:41.218497 (+ 2004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:41.219216 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:41.219238 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:41.219844 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:41.220623 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:41.220644 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:41.222115 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:41.222118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:41.223540 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:41.223555 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:41.224345 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:41.224349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:41.224594 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:41.225326 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:41.225343 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:41.226969 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:41.229017 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:41.229026 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:41.229049 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:41.229465 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:41.229757 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:41.229761 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:41.229763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:41.229961 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:41.230401 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:41.230410 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:41.230778 (+ 368us) client_negotiation.cc:769] Sending connection context
0619 18:42:41.231013 (+ 235us) client_negotiation.cc:240] Negotiation successful
0619 18:42:41.231445 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":127,"threads_started":1}
I20260619 18:42:41.233851 25460 heartbeater.cc:347] Connected to a master server at 127.21.143.126:42293
I20260619 18:42:41.234156 25460 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.67@KRBTEST.COM" created.
I20260619 18:42:41.234750 25460 heartbeater.cc:511] Master 127.21.143.126:42293 requested a full tablet report, sending...
I20260619 18:42:41.235886 25135 ts_manager.cc:194] Registered new tserver with Master: 4c123c835f7a43c69aa6f083c984536c (127.21.143.66:42707)
I20260619 18:42:41.236650 25135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.66@KRBTEST.COM'} at 127.21.143.66:41987
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab.
Entry for principal kudu/127.21.143.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab.
I20260619 18:42:41.250504 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.67:0
--local_ip_for_outbound_sockets=127.21.143.67
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:42293
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:42691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:41.359661 25465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:41.359902 25465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:41.359989 25465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:41.364204 25465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:41.364262 25465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:41.364353 25465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.67
I20260619 18:42:41.369597 25465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:42691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:42293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.67
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:41.370636 25465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:41.371729 25465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:41.382447 25470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:41.382402 25473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:41.382411 25471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:41.382634 25465 server_base.cc:1034] running on GCE node
I20260619 18:42:41.383045 25465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:41.383684 25465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:41.384850 25465 hybrid_clock.cc:648] HybridClock initialized: now 1781894561384837 us; error 24 us; skew 500 ppm
Jun 19 18:42:41 dist-test-slave-mld5 krb5kdc[25090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894561, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:41.387557 25465 init.cc:377] Logged in from keytab as kudu/127.21.143.67@KRBTEST.COM (short username kudu)
I20260619 18:42:41.388393 25465 webserver.cc:460] Webserver started at http://127.21.143.67:33949/ using document root <none> and password file <none>
I20260619 18:42:41.388958 25465 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:41.389026 25465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:41.389236 25465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:41.391215 25465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-2/data/instance:
uuid: "f4e026ba18fd4b32a918fbd3b84b071c"
format_stamp: "Formatted at 2026-06-19 18:42:41 on dist-test-slave-mld5"
server_key: "5c12651542e97156893385b917de2f85"
server_key_iv: "7bbc7d0cd699ac9c0c0ca762b4835184"
server_key_version: "encryptionkey@0"
I20260619 18:42:41.391832 25465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance:
uuid: "f4e026ba18fd4b32a918fbd3b84b071c"
format_stamp: "Formatted at 2026-06-19 18:42:41 on dist-test-slave-mld5"
server_key: "5c12651542e97156893385b917de2f85"
server_key_iv: "7bbc7d0cd699ac9c0c0ca762b4835184"
server_key_version: "encryptionkey@0"
I20260619 18:42:41.395412 25465 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:42:41.397727 25480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:41.398977 25465 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:42:41.399129 25465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-2/wal
uuid: "f4e026ba18fd4b32a918fbd3b84b071c"
format_stamp: "Formatted at 2026-06-19 18:42:41 on dist-test-slave-mld5"
server_key: "5c12651542e97156893385b917de2f85"
server_key_iv: "7bbc7d0cd699ac9c0c0ca762b4835184"
server_key_version: "encryptionkey@0"
I20260619 18:42:41.399237 25465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:41.417816 25465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:41.421250 25465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:41.421456 25465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:41.422049 25465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:41.422864 25465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:41.422935 25465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:41.423012 25465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:41.423062 25465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:41.431735 25465 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.67:42991
I20260619 18:42:41.431758 25593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.67:42991 every 8 connection(s)
I20260619 18:42:41.432623 25465 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
I20260619 18:42:41.436403 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 25465
I20260619 18:42:41.436553 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance
I20260619 18:42:41.436867 22077 external_mini_cluster.cc:1428] Setting key 76384f3f68c35b7ca319af933df405af
Jun 19 18:42:41 dist-test-slave-mld5 krb5kdc[25090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894561, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:41.446794 25174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:41.434718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:55061 (local address 127.21.143.126:42293)
0619 18:42:41.434860 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:41.434869 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:41.435677 (+ 808us) server_negotiation.cc:407] Connection header received
0619 18:42:41.437469 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:41.437473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:41.437519 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:41.437616 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:41.439128 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:41.439721 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:41.440818 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:41.441021 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:41.441093 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:41.443482 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:41.443485 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:41.443487 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:41.443509 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:41.444708 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:41.445270 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:41.445275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:41.445278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:41.445361 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:41.445657 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:41.445659 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:41.445661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:41.445824 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:41.445911 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:41.446505 (+ 594us) server_negotiation.cc:299] Negotiation successful
0619 18:42:41.446628 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 18:42:41.447718 25596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:41.435014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42293 (local address 127.21.143.67:55061)
0619 18:42:41.435527 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:41.435570 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:41.437207 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:41.437829 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:41.437840 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:41.438370 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:41.438977 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:41.439001 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:41.439824 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:41.439827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:41.440718 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:41.440727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:41.441142 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:41.441144 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:41.441287 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:41.441889 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:41.441905 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:41.443281 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:41.444804 (+ 1523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:41.444809 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:41.444821 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:41.445174 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:41.445435 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:41.445437 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:41.445438 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:41.445571 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:41.445937 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:41.445942 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:41.446246 (+ 304us) client_negotiation.cc:769] Sending connection context
0619 18:42:41.446445 (+ 199us) client_negotiation.cc:240] Negotiation successful
0619 18:42:41.446777 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":106,"threads_started":1}
I20260619 18:42:41.448812 25594 heartbeater.cc:347] Connected to a master server at 127.21.143.126:42293
I20260619 18:42:41.449065 25594 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:41.449559 25594 heartbeater.cc:511] Master 127.21.143.126:42293 requested a full tablet report, sending...
I20260619 18:42:41.450603 25134 ts_manager.cc:194] Registered new tserver with Master: f4e026ba18fd4b32a918fbd3b84b071c (127.21.143.67:42991)
I20260619 18:42:41.451185 25134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.67@KRBTEST.COM'} at 127.21.143.67:55061
I20260619 18:42:41.462500 22077 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 18:42:41 dist-test-slave-mld5 krb5kdc[25090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894561, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 18:42:41 dist-test-slave-mld5 krb5kdc[25090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894561, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:41.487543 25174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:41.479416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37122 (local address 127.21.143.126:42293)
0619 18:42:41.479567 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:41.479572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:41.479741 (+ 169us) server_negotiation.cc:407] Connection header received
0619 18:42:41.479894 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:41.479898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:41.479948 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:41.480043 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:41.480754 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:41.481277 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:41.482101 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:41.482318 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:41.482425 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:41.484748 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:41.484757 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:41.484759 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:41.484781 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:41.485938 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:41.486330 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:41.486335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:41.486337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:41.486412 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:41.486680 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:41.486685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:41.486687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:41.486893 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:41.487032 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:41.487253 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 18:42:41.487388 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 18:42:41.487886 25134 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37122
I20260619 18:42:41.488647 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 25197
I20260619 18:42:41.493997 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 25331
I20260619 18:42:41.499396 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 25465
I20260619 18:42:41.504390 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 25103
2026-06-19T18:42:41Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2105 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:42:41 dist-test-slave-mld5 krb5kdc[25606](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:42:41 dist-test-slave-mld5 krb5kdc[25606](info): set up 2 sockets
Jun 19 18:42:41 dist-test-slave-mld5 krb5kdc[25606](info): commencing operation
krb5kdc: starting...
W20260619 18:42:42.539589 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:42:42 dist-test-slave-mld5 krb5kdc[25606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:42:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:42:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.143.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
I20260619 18:42:42.658370 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:40649
--webserver_interface=127.21.143.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:33849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:40649
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:42.765400 25619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:42.765705 25619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:42.765794 25619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:42.769855 25619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:42:42.769937 25619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:42.769975 25619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:42.770013 25619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:42:42.770053 25619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:42:42.775115 25619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:33849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:40649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:40649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:42.776168 25619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:42.777180 25619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:42.785420 25627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:42.785455 25625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:42.785416 25624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:42.785676 25619 server_base.cc:1034] running on GCE node
I20260619 18:42:42.786248 25619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:42.787403 25619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:42.788607 25619 hybrid_clock.cc:648] HybridClock initialized: now 1781894562788587 us; error 40 us; skew 500 ppm
Jun 19 18:42:42 dist-test-slave-mld5 krb5kdc[25606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894562, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:42.794915 25619 init.cc:377] Logged in from keytab as kudu/127.21.143.126@KRBTEST.COM (short username kudu)
I20260619 18:42:42.795930 25619 webserver.cc:460] Webserver started at http://127.21.143.126:38263/ using document root <none> and password file <none>
I20260619 18:42:42.796499 25619 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:42.796592 25619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:42.796834 25619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:42.798887 25619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "27ddcf6db52547179461b90094b7372e"
format_stamp: "Formatted at 2026-06-19 18:42:42 on dist-test-slave-mld5"
server_key: "b2ce04b8c9aa97888f4a95fe6747cd7e"
server_key_iv: "72fb52fd0e170d96b6d5e0c58268c4d7"
server_key_version: "encryptionkey@0"
I20260619 18:42:42.799518 25619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "27ddcf6db52547179461b90094b7372e"
format_stamp: "Formatted at 2026-06-19 18:42:42 on dist-test-slave-mld5"
server_key: "b2ce04b8c9aa97888f4a95fe6747cd7e"
server_key_iv: "72fb52fd0e170d96b6d5e0c58268c4d7"
server_key_version: "encryptionkey@0"
I20260619 18:42:42.803117 25619 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:42:42.805521 25634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:42.806699 25619 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:42:42.806845 25619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "27ddcf6db52547179461b90094b7372e"
format_stamp: "Formatted at 2026-06-19 18:42:42 on dist-test-slave-mld5"
server_key: "b2ce04b8c9aa97888f4a95fe6747cd7e"
server_key_iv: "72fb52fd0e170d96b6d5e0c58268c4d7"
server_key_version: "encryptionkey@0"
I20260619 18:42:42.806951 25619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:42.819554 25619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:42.822752 25619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:42.822947 25619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:42.829782 25619 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:40649
I20260619 18:42:42.829804 25686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:40649 every 8 connection(s)
I20260619 18:42:42.830688 25619 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:42:42.833259 25687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:42.834343 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 25619
I20260619 18:42:42.834473 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:42:42.834774 22077 external_mini_cluster.cc:1428] Setting key 98e42e92e380bda2a560bfd44d6de754
I20260619 18:42:42.839102 25687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27ddcf6db52547179461b90094b7372e: Bootstrap starting.
I20260619 18:42:42.841292 25687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27ddcf6db52547179461b90094b7372e: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:42.842140 25687 log.cc:824] T 00000000000000000000000000000000 P 27ddcf6db52547179461b90094b7372e: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:42:42 dist-test-slave-mld5 krb5kdc[25606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:42.844432 25687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27ddcf6db52547179461b90094b7372e: No bootstrap required, opened a new log
I20260619 18:42:42.848603 25687 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27ddcf6db52547179461b90094b7372e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ddcf6db52547179461b90094b7372e" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 40649 } }
I20260619 18:42:42.848766 25690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:42.836156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34956 (local address 127.21.143.126:40649)
0619 18:42:42.836577 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:42.836591 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:42.836625 (+ 34us) server_negotiation.cc:407] Connection header received
0619 18:42:42.837245 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:42.837260 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:42.837567 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:42.837899 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:42.838749 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:42.839817 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:42.840666 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:42.841129 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:42.841309 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:42.844120 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:42.844126 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:42.844140 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:42.844167 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:42.845772 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:42.846276 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:42.846280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:42.846285 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:42.846390 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:42.846698 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:42.846700 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:42.846701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:42.847066 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:42.847205 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:42.847553 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 18:42:42.847868 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":123,"threads_started":1}
I20260619 18:42:42.848819 25687 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27ddcf6db52547179461b90094b7372e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:42.848886 25687 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27ddcf6db52547179461b90094b7372e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27ddcf6db52547179461b90094b7372e, State: Initialized, Role: FOLLOWER
I20260619 18:42:42.849329 25687 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27ddcf6db52547179461b90094b7372e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ddcf6db52547179461b90094b7372e" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 40649 } }
I20260619 18:42:42.849450 25687 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27ddcf6db52547179461b90094b7372e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:42:42.849555 25687 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27ddcf6db52547179461b90094b7372e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:42:42.849663 25687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27ddcf6db52547179461b90094b7372e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:42.850518 25687 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27ddcf6db52547179461b90094b7372e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ddcf6db52547179461b90094b7372e" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 40649 } }
I20260619 18:42:42.850862 25687 leader_election.cc:302] T 00000000000000000000000000000000 P 27ddcf6db52547179461b90094b7372e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27ddcf6db52547179461b90094b7372e; no voters:
I20260619 18:42:42.851187 25687 leader_election.cc:288] T 00000000000000000000000000000000 P 27ddcf6db52547179461b90094b7372e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:42:42.851292 25692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27ddcf6db52547179461b90094b7372e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:42.851542 25692 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27ddcf6db52547179461b90094b7372e [term 1 LEADER]: Becoming Leader. State: Replica: 27ddcf6db52547179461b90094b7372e, State: Running, Role: LEADER
I20260619 18:42:42.851890 25692 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27ddcf6db52547179461b90094b7372e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ddcf6db52547179461b90094b7372e" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 40649 } }
I20260619 18:42:42.852416 25687 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27ddcf6db52547179461b90094b7372e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:42:42.853459 25694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27ddcf6db52547179461b90094b7372e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27ddcf6db52547179461b90094b7372e. Latest consensus state: current_term: 1 leader_uuid: "27ddcf6db52547179461b90094b7372e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ddcf6db52547179461b90094b7372e" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 40649 } } }
I20260619 18:42:42.853579 25694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27ddcf6db52547179461b90094b7372e [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:42.853955 25701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:42:42.853921 25693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27ddcf6db52547179461b90094b7372e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27ddcf6db52547179461b90094b7372e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ddcf6db52547179461b90094b7372e" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 40649 } } }
I20260619 18:42:42.854056 25693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27ddcf6db52547179461b90094b7372e [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:42.857057 25701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:42:42.862139 25701 catalog_manager.cc:1347] Generated new cluster ID: aaae5f0dcc164570ab1f90bfb04bad60
I20260619 18:42:42.862207 25701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:42:42.868158 25701 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:42:42.869021 25701 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:42:42.879277 25701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27ddcf6db52547179461b90094b7372e: Generated new TSK 0
I20260619 18:42:42.880033 25701 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.143.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab.
Entry for principal kudu/127.21.143.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab.
I20260619 18:42:42.912521 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.65:0
--local_ip_for_outbound_sockets=127.21.143.65
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:40649
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:33849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:43.020586 25713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:43.020856 25713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:43.020929 25713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:43.025013 25713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:43.025092 25713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:43.025183 25713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.65
I20260619 18:42:43.030246 25713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:33849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:40649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.65
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:43.031209 25713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:43.032229 25713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:43.041256 25719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:43.041257 25721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:43.041280 25718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:43.041458 25713 server_base.cc:1034] running on GCE node
I20260619 18:42:43.041862 25713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:43.042433 25713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:43.043584 25713 hybrid_clock.cc:648] HybridClock initialized: now 1781894563043565 us; error 30 us; skew 500 ppm
Jun 19 18:42:43 dist-test-slave-mld5 krb5kdc[25606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894563, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:43.046132 25713 init.cc:377] Logged in from keytab as kudu/127.21.143.65@KRBTEST.COM (short username kudu)
I20260619 18:42:43.046948 25713 webserver.cc:460] Webserver started at http://127.21.143.65:37711/ using document root <none> and password file <none>
I20260619 18:42:43.047582 25713 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:43.047667 25713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:43.047885 25713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:43.049945 25713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-0/data/instance:
uuid: "b4ddc23037fc412ca4c35a3d8b7016ff"
format_stamp: "Formatted at 2026-06-19 18:42:43 on dist-test-slave-mld5"
server_key: "3ea755676017644c2914511a6fa55018"
server_key_iv: "2e94ade1a2a9ce103d1eca789855311f"
server_key_version: "encryptionkey@0"
I20260619 18:42:43.050518 25713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance:
uuid: "b4ddc23037fc412ca4c35a3d8b7016ff"
format_stamp: "Formatted at 2026-06-19 18:42:43 on dist-test-slave-mld5"
server_key: "3ea755676017644c2914511a6fa55018"
server_key_iv: "2e94ade1a2a9ce103d1eca789855311f"
server_key_version: "encryptionkey@0"
I20260619 18:42:43.054201 25713 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 18:42:43.056566 25728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:43.057682 25713 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:42:43.057811 25713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-0/wal
uuid: "b4ddc23037fc412ca4c35a3d8b7016ff"
format_stamp: "Formatted at 2026-06-19 18:42:43 on dist-test-slave-mld5"
server_key: "3ea755676017644c2914511a6fa55018"
server_key_iv: "2e94ade1a2a9ce103d1eca789855311f"
server_key_version: "encryptionkey@0"
I20260619 18:42:43.057914 25713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:43.069932 25713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:43.072475 25713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:43.072671 25713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:43.073204 25713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:43.073994 25713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:43.074076 25713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:43.074141 25713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:43.074195 25713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:43.082984 25713 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.65:40501
I20260619 18:42:43.083003 25841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.65:40501 every 8 connection(s)
I20260619 18:42:43.083863 25713 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
I20260619 18:42:43.088141 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 25713
I20260619 18:42:43.088266 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance
I20260619 18:42:43.088557 22077 external_mini_cluster.cc:1428] Setting key 148d7f4d4a3d4e66033e7b30458f7a32
Jun 19 18:42:43 dist-test-slave-mld5 krb5kdc[25606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894563, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.143.66@KRBTEST.COM; defaulting to no policy
I20260619 18:42:43.097434 25690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:43.085541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:41899 (local address 127.21.143.126:40649)
0619 18:42:43.085672 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:43.085675 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:43.086426 (+ 751us) server_negotiation.cc:407] Connection header received
0619 18:42:43.087847 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:43.087850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:43.087909 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:43.088016 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:43.089340 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:43.089958 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:43.091061 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:43.091351 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:43.091467 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:43.093886 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:43.093889 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:43.093891 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:43.093913 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:43.095455 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:43.095955 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:43.095960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:43.095962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:43.096042 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:43.096374 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:43.096379 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:43.096380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:43.096552 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:43.096656 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:43.097129 (+ 473us) server_negotiation.cc:299] Negotiation successful
0619 18:42:43.097268 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 18:42:43.098136 25844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:43.085852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40649 (local address 127.21.143.65:41899)
0619 18:42:43.086282 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:43.086323 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:43.087641 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:43.088218 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:43.088229 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:43.088643 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:43.089188 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:43.089208 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:43.090075 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:43.090078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:43.090937 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:43.090946 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:43.091498 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:43.091502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:43.091672 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:43.092282 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:43.092299 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:43.093718 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:43.095582 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:43.095587 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:43.095600 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:43.095851 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:43.096140 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:43.096143 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:43.096145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:43.096278 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:43.096666 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:43.096674 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:43.096902 (+ 228us) client_negotiation.cc:769] Sending connection context
0619 18:42:43.097087 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 18:42:43.097380 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":110,"threads_started":1}
I20260619 18:42:43.099277 25842 heartbeater.cc:347] Connected to a master server at 127.21.143.126:40649
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.66@KRBTEST.COM" created.
I20260619 18:42:43.099578 25842 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:43.100109 25842 heartbeater.cc:511] Master 127.21.143.126:40649 requested a full tablet report, sending...
I20260619 18:42:43.101577 25651 ts_manager.cc:194] Registered new tserver with Master: b4ddc23037fc412ca4c35a3d8b7016ff (127.21.143.65:40501)
I20260619 18:42:43.102632 25651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.65@KRBTEST.COM'} at 127.21.143.65:41899
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab.
Entry for principal kudu/127.21.143.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab.
I20260619 18:42:43.114194 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.66:0
--local_ip_for_outbound_sockets=127.21.143.66
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:40649
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:33849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:43.217463 25847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:43.217764 25847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:43.217839 25847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:43.221845 25847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:43.221936 25847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:43.222029 25847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.66
I20260619 18:42:43.227787 25847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:33849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:40649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.66
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:43.228940 25847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:43.230043 25847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:43.239599 25853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:43.239614 25852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:43.239784 25847 server_base.cc:1034] running on GCE node
W20260619 18:42:43.239679 25855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:43.240302 25847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:43.240916 25847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:43.242070 25847 hybrid_clock.cc:648] HybridClock initialized: now 1781894563242052 us; error 29 us; skew 500 ppm
Jun 19 18:42:43 dist-test-slave-mld5 krb5kdc[25606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894563, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:43.244776 25847 init.cc:377] Logged in from keytab as kudu/127.21.143.66@KRBTEST.COM (short username kudu)
I20260619 18:42:43.245558 25847 webserver.cc:460] Webserver started at http://127.21.143.66:35423/ using document root <none> and password file <none>
I20260619 18:42:43.246058 25847 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:43.246129 25847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:43.246299 25847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:43.248301 25847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-1/data/instance:
uuid: "e8c1ca3e7c9148708cb2e12b76cb25d1"
format_stamp: "Formatted at 2026-06-19 18:42:43 on dist-test-slave-mld5"
server_key: "b29ee00452b57dd609acd782f5404105"
server_key_iv: "8d51f88912061e8b0c8eec2c10fb3ba1"
server_key_version: "encryptionkey@0"
I20260619 18:42:43.248914 25847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance:
uuid: "e8c1ca3e7c9148708cb2e12b76cb25d1"
format_stamp: "Formatted at 2026-06-19 18:42:43 on dist-test-slave-mld5"
server_key: "b29ee00452b57dd609acd782f5404105"
server_key_iv: "8d51f88912061e8b0c8eec2c10fb3ba1"
server_key_version: "encryptionkey@0"
I20260619 18:42:43.252614 25847 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:42:43.255055 25862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:43.256342 25847 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:42:43.256460 25847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-1/wal
uuid: "e8c1ca3e7c9148708cb2e12b76cb25d1"
format_stamp: "Formatted at 2026-06-19 18:42:43 on dist-test-slave-mld5"
server_key: "b29ee00452b57dd609acd782f5404105"
server_key_iv: "8d51f88912061e8b0c8eec2c10fb3ba1"
server_key_version: "encryptionkey@0"
I20260619 18:42:43.256565 25847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:43.275667 25847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:43.279116 25847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:43.279390 25847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:43.280174 25847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:43.281342 25847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:43.281409 25847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:43.281468 25847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:43.281504 25847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:43.291783 25847 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.66:45925
I20260619 18:42:43.291807 25975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.66:45925 every 8 connection(s)
I20260619 18:42:43.292721 25847 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
I20260619 18:42:43.299933 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 25847
I20260619 18:42:43.300072 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance
I20260619 18:42:43.300385 22077 external_mini_cluster.cc:1428] Setting key 98b4ca2e789f57fc2386fda8df6a6b2f
Jun 19 18:42:43 dist-test-slave-mld5 krb5kdc[25606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894563, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:43.308804 25690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:43.294525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:53165 (local address 127.21.143.126:40649)
0619 18:42:43.294668 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:43.294673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:43.295635 (+ 962us) server_negotiation.cc:407] Connection header received
0619 18:42:43.297246 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:43.297250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:43.297310 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:43.297393 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:43.298714 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:43.299280 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:43.300914 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:43.301410 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:43.301536 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:43.304747 (+ 3211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:43.304751 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:43.304753 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:43.304779 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:43.306383 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:43.306931 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:43.306935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:43.306937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:43.307028 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:43.307480 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:43.307486 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:43.307490 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:43.307791 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:43.307914 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:43.308436 (+ 522us) server_negotiation.cc:299] Negotiation successful
0619 18:42:43.308582 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 18:42:43.309684 25978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:43.294937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40649 (local address 127.21.143.66:53165)
0619 18:42:43.295482 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:43.295526 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:43.296993 (+ 1467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:43.297580 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:43.297588 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:43.298047 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:43.298550 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:43.298563 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:43.299436 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:43.299440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:43.300802 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:43.300812 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:43.301571 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:43.301575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:43.301758 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:43.302370 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:43.302391 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:43.304537 (+ 2146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:43.306525 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:43.306531 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:43.306547 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:43.306822 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:43.307131 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:43.307135 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:43.307136 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:43.307285 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:43.307912 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:43.307917 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:43.308185 (+ 268us) client_negotiation.cc:769] Sending connection context
0619 18:42:43.308349 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 18:42:43.308616 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":129,"threads_started":1}
WARNING: no policy specified for kudu/127.21.143.67@KRBTEST.COM; defaulting to no policy
I20260619 18:42:43.310950 25976 heartbeater.cc:347] Connected to a master server at 127.21.143.126:40649
I20260619 18:42:43.311237 25976 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:43.311776 25976 heartbeater.cc:511] Master 127.21.143.126:40649 requested a full tablet report, sending...
I20260619 18:42:43.312839 25651 ts_manager.cc:194] Registered new tserver with Master: e8c1ca3e7c9148708cb2e12b76cb25d1 (127.21.143.66:45925)
I20260619 18:42:43.313519 25651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.66@KRBTEST.COM'} at 127.21.143.66:53165
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.67@KRBTEST.COM" created.
I20260619 18:42:44.106065 25842 heartbeater.cc:503] Master 127.21.143.126:40649 was elected leader, sending a full tablet report...
I20260619 18:42:44.315574 25976 heartbeater.cc:503] Master 127.21.143.126:40649 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab.
Entry for principal kudu/127.21.143.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab.
W20260619 18:42:44.357578 22077 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.21.143.67: real 1.055s user 0.000s sys 0.004s
I20260619 18:42:44.358060 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.67:0
--local_ip_for_outbound_sockets=127.21.143.67
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:40649
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:33849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:44.478223 25981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:44.478518 25981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:44.478561 25981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:44.483336 25981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:44.483407 25981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:44.483493 25981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.67
I20260619 18:42:44.489233 25981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:33849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:40649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.67
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:44.490350 25981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:44.491456 25981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:44.500975 25987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:44.500972 25989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:44.500967 25986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:44.501365 25981 server_base.cc:1034] running on GCE node
I20260619 18:42:44.501845 25981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:44.502480 25981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:44.503659 25981 hybrid_clock.cc:648] HybridClock initialized: now 1781894564503641 us; error 28 us; skew 500 ppm
Jun 19 18:42:44 dist-test-slave-mld5 krb5kdc[25606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894564, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:44.506443 25981 init.cc:377] Logged in from keytab as kudu/127.21.143.67@KRBTEST.COM (short username kudu)
I20260619 18:42:44.507195 25981 webserver.cc:460] Webserver started at http://127.21.143.67:46277/ using document root <none> and password file <none>
I20260619 18:42:44.507769 25981 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:44.507829 25981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:44.507989 25981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:44.509943 25981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-2/data/instance:
uuid: "fb946ce780444475ba2e52316f92f84b"
format_stamp: "Formatted at 2026-06-19 18:42:44 on dist-test-slave-mld5"
server_key: "ea396c26fa6c5e7f48826a9642754262"
server_key_iv: "136bc03c5b85beb0f313a46bb5dae7c4"
server_key_version: "encryptionkey@0"
I20260619 18:42:44.510584 25981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance:
uuid: "fb946ce780444475ba2e52316f92f84b"
format_stamp: "Formatted at 2026-06-19 18:42:44 on dist-test-slave-mld5"
server_key: "ea396c26fa6c5e7f48826a9642754262"
server_key_iv: "136bc03c5b85beb0f313a46bb5dae7c4"
server_key_version: "encryptionkey@0"
I20260619 18:42:44.514264 25981 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 18:42:44.516746 25996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:44.517992 25981 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:42:44.518113 25981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-2/wal
uuid: "fb946ce780444475ba2e52316f92f84b"
format_stamp: "Formatted at 2026-06-19 18:42:44 on dist-test-slave-mld5"
server_key: "ea396c26fa6c5e7f48826a9642754262"
server_key_iv: "136bc03c5b85beb0f313a46bb5dae7c4"
server_key_version: "encryptionkey@0"
I20260619 18:42:44.518217 25981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:44.539888 25981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:44.543406 25981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:44.543614 25981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:44.544225 25981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:44.545123 25981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:44.545169 25981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:44.545238 25981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:44.545269 25981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:44.555588 25981 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.67:35763
I20260619 18:42:44.555605 26109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.67:35763 every 8 connection(s)
I20260619 18:42:44.556490 25981 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
I20260619 18:42:44.565048 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 25981
I20260619 18:42:44.565189 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance
I20260619 18:42:44.565479 22077 external_mini_cluster.cc:1428] Setting key c013460cd046745562a840bc685f6848
Jun 19 18:42:44 dist-test-slave-mld5 krb5kdc[25606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894564, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:44.573544 26112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:44.558277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:51291 (local address 127.21.143.126:40649)
0619 18:42:44.558529 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:44.558533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:44.561753 (+ 3220us) server_negotiation.cc:407] Connection header received
0619 18:42:44.561823 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:44.561828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:44.561909 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:44.562004 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:44.563921 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:44.564735 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:44.566443 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:44.566777 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:44.566886 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:44.570009 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:44.570013 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:44.570015 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:44.570040 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:44.571632 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:44.572121 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:44.572124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:44.572126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:44.572202 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:44.572527 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:44.572530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:44.572531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:44.572701 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:44.572773 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:44.573276 (+ 503us) server_negotiation.cc:299] Negotiation successful
0619 18:42:44.573415 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":91,"threads_started":1}
I20260619 18:42:44.574363 26113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:44.558723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40649 (local address 127.21.143.67:51291)
0619 18:42:44.559358 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:44.559424 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:44.561430 (+ 2006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:44.562261 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:44.562273 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:44.562906 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:44.563739 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:44.563763 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:44.564860 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:44.564865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:44.566286 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:44.566297 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:44.567063 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:44.567069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:44.567282 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:44.568009 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:44.568030 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:44.569845 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:44.571743 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:44.571749 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:44.571767 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:44.572031 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:44.572277 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:44.572279 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:44.572280 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:44.572444 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:44.572784 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:44.572789 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:44.573022 (+ 233us) client_negotiation.cc:769] Sending connection context
0619 18:42:44.573220 (+ 198us) client_negotiation.cc:240] Negotiation successful
0619 18:42:44.573483 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":168,"threads_started":1}
I20260619 18:42:44.575519 26110 heartbeater.cc:347] Connected to a master server at 127.21.143.126:40649
I20260619 18:42:44.575723 26110 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:44.576191 26110 heartbeater.cc:511] Master 127.21.143.126:40649 requested a full tablet report, sending...
I20260619 18:42:44.577126 25651 ts_manager.cc:194] Registered new tserver with Master: fb946ce780444475ba2e52316f92f84b (127.21.143.67:35763)
I20260619 18:42:44.577705 25651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.67@KRBTEST.COM'} at 127.21.143.67:51291
I20260619 18:42:44.580293 22077 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 18:42:44 dist-test-slave-mld5 krb5kdc[25606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 18:42:44 dist-test-slave-mld5 krb5kdc[25606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.65@KRBTEST.COM
I20260619 18:42:44.663147 26118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:44.652916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34512 (local address 127.21.143.65:40501)
0619 18:42:44.653411 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:44.653417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:44.653440 (+ 23us) server_negotiation.cc:407] Connection header received
0619 18:42:44.653489 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:44.653494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:44.653811 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:44.653940 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:44.654919 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:44.655768 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:44.656962 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:44.657334 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:44.657419 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:44.659808 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:44.659817 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:44.659831 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:44.659880 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:44.661574 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:44.661981 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:44.661986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:44.661987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:44.662063 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:44.662285 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:44.662287 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:44.662288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:44.662559 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:44.662727 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:44.662874 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 18:42:44.663003 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":64,"threads_started":1}
Jun 19 18:42:44 dist-test-slave-mld5 krb5kdc[25606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:44.674269 26112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:44.666062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34972 (local address 127.21.143.126:40649)
0619 18:42:44.666200 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:44.666207 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:44.666292 (+ 85us) server_negotiation.cc:407] Connection header received
0619 18:42:44.666448 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:44.666452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:44.666504 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:44.666591 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:44.667171 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:44.667894 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:44.668678 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:44.668974 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:44.669084 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:44.671399 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:44.671402 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:44.671404 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:44.671425 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:44.672813 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:44.673210 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:44.673212 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:44.673213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:44.673286 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:44.673501 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:44.673503 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:44.673504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:44.673700 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:44.673812 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:44.674054 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 18:42:44.674153 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 18:42:44.674677 25651 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34972
I20260619 18:42:44.675346 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 25713
I20260619 18:42:44.682408 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 25847
I20260619 18:42:44.688056 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 25981
I20260619 18:42:44.693423 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 25619
2026-06-19T18:42:44Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3189 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:42:46 dist-test-slave-mld5 krb5kdc[26123](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:42:46 dist-test-slave-mld5 krb5kdc[26123](info): set up 2 sockets
Jun 19 18:42:46 dist-test-slave-mld5 krb5kdc[26123](info): commencing operation
krb5kdc: starting...
W20260619 18:42:47.327478 22077 mini_kdc.cc:121] Time spent starting KDC: real 2.612s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:42:47 dist-test-slave-mld5 krb5kdc[26123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:42:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:42:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.143.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
I20260619 18:42:47.501399 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:39435
--webserver_interface=127.21.143.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:34973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:39435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:47.616415 26136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:47.616721 26136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:47.616767 26136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:47.620963 26136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:42:47.621024 26136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:47.621047 26136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:47.621063 26136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:42:47.621079 26136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:42:47.626492 26136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:34973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:39435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:39435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:47.627737 26136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:47.628813 26136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:47.637301 26144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:47.637343 26142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:47.637318 26141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:47.637485 26136 server_base.cc:1034] running on GCE node
I20260619 18:42:47.638165 26136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:47.639231 26136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:47.640419 26136 hybrid_clock.cc:648] HybridClock initialized: now 1781894567640401 us; error 29 us; skew 500 ppm
Jun 19 18:42:47 dist-test-slave-mld5 krb5kdc[26123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894567, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:47.643075 26136 init.cc:377] Logged in from keytab as kudu/127.21.143.126@KRBTEST.COM (short username kudu)
I20260619 18:42:47.643865 26136 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:47.643955 26136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:47.644199 26136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:47.646216 26136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "3cdd06f76e50434d953e5d73046e0b32"
format_stamp: "Formatted at 2026-06-19 18:42:47 on dist-test-slave-mld5"
server_key: "2e7c75ad2e358da3155d4f3bee4e4fee"
server_key_iv: "87f2613d7a77e783af24812eb93c56d6"
server_key_version: "encryptionkey@0"
I20260619 18:42:47.646788 26136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "3cdd06f76e50434d953e5d73046e0b32"
format_stamp: "Formatted at 2026-06-19 18:42:47 on dist-test-slave-mld5"
server_key: "2e7c75ad2e358da3155d4f3bee4e4fee"
server_key_iv: "87f2613d7a77e783af24812eb93c56d6"
server_key_version: "encryptionkey@0"
I20260619 18:42:47.650524 26136 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.004s
I20260619 18:42:47.653028 26150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:47.654259 26136 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:42:47.654408 26136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "3cdd06f76e50434d953e5d73046e0b32"
format_stamp: "Formatted at 2026-06-19 18:42:47 on dist-test-slave-mld5"
server_key: "2e7c75ad2e358da3155d4f3bee4e4fee"
server_key_iv: "87f2613d7a77e783af24812eb93c56d6"
server_key_version: "encryptionkey@0"
I20260619 18:42:47.654520 26136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:47.690088 26136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:47.692889 26136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:47.693096 26136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:47.699920 26136 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:39435
I20260619 18:42:47.700235 26202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:39435 every 8 connection(s)
I20260619 18:42:47.700610 26136 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:42:47.703325 26203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:47.707798 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 26136
I20260619 18:42:47.707927 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:42:47.708215 22077 external_mini_cluster.cc:1428] Setting key 04565f87041fa7893f776511c46465c4
I20260619 18:42:47.709028 26203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cdd06f76e50434d953e5d73046e0b32: Bootstrap starting.
I20260619 18:42:47.711270 26203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cdd06f76e50434d953e5d73046e0b32: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:47.712002 26203 log.cc:824] T 00000000000000000000000000000000 P 3cdd06f76e50434d953e5d73046e0b32: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:47.713747 26203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cdd06f76e50434d953e5d73046e0b32: No bootstrap required, opened a new log
Jun 19 18:42:47 dist-test-slave-mld5 krb5kdc[26123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:47.717991 26203 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cdd06f76e50434d953e5d73046e0b32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cdd06f76e50434d953e5d73046e0b32" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39435 } }
I20260619 18:42:47.718277 26203 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cdd06f76e50434d953e5d73046e0b32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:47.718348 26203 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cdd06f76e50434d953e5d73046e0b32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cdd06f76e50434d953e5d73046e0b32, State: Initialized, Role: FOLLOWER
I20260619 18:42:47.718835 26203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cdd06f76e50434d953e5d73046e0b32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cdd06f76e50434d953e5d73046e0b32" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39435 } }
I20260619 18:42:47.718983 26203 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cdd06f76e50434d953e5d73046e0b32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:42:47.719053 26203 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cdd06f76e50434d953e5d73046e0b32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:42:47.719147 26203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cdd06f76e50434d953e5d73046e0b32 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:47.719959 26203 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cdd06f76e50434d953e5d73046e0b32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cdd06f76e50434d953e5d73046e0b32" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39435 } }
I20260619 18:42:47.720304 26203 leader_election.cc:302] T 00000000000000000000000000000000 P 3cdd06f76e50434d953e5d73046e0b32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cdd06f76e50434d953e5d73046e0b32; no voters:
I20260619 18:42:47.720592 26203 leader_election.cc:288] T 00000000000000000000000000000000 P 3cdd06f76e50434d953e5d73046e0b32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:42:47.720701 26208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cdd06f76e50434d953e5d73046e0b32 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:47.720903 26208 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cdd06f76e50434d953e5d73046e0b32 [term 1 LEADER]: Becoming Leader. State: Replica: 3cdd06f76e50434d953e5d73046e0b32, State: Running, Role: LEADER
I20260619 18:42:47.721220 26208 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cdd06f76e50434d953e5d73046e0b32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cdd06f76e50434d953e5d73046e0b32" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39435 } }
I20260619 18:42:47.721832 26203 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cdd06f76e50434d953e5d73046e0b32 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:42:47.722411 26208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cdd06f76e50434d953e5d73046e0b32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cdd06f76e50434d953e5d73046e0b32. Latest consensus state: current_term: 1 leader_uuid: "3cdd06f76e50434d953e5d73046e0b32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cdd06f76e50434d953e5d73046e0b32" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39435 } } }
I20260619 18:42:47.722543 26208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cdd06f76e50434d953e5d73046e0b32 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:47.722958 26209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cdd06f76e50434d953e5d73046e0b32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cdd06f76e50434d953e5d73046e0b32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cdd06f76e50434d953e5d73046e0b32" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39435 } } }
I20260619 18:42:47.723045 26206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:47.709589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40928 (local address 127.21.143.126:39435)
0619 18:42:47.709979 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:47.709994 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:47.710028 (+ 34us) server_negotiation.cc:407] Connection header received
0619 18:42:47.710630 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:47.710646 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:47.710955 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:47.711292 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:47.712151 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:47.713072 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:47.713991 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:47.714688 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:47.714863 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:47.717419 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:47.717426 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:47.717443 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:47.717470 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:47.719412 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:47.719838 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:47.719844 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:47.719849 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:47.719963 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:47.720262 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:47.720265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:47.720267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:47.720801 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:47.721635 (+ 834us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:47.721777 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 18:42:47.722047 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":110,"threads_started":1}
I20260619 18:42:47.723399 26215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:42:47.723601 26209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cdd06f76e50434d953e5d73046e0b32 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:47.726481 26215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:42:47.732400 26215 catalog_manager.cc:1347] Generated new cluster ID: 5880c9e88c9048b293c70e621fed33ef
I20260619 18:42:47.732483 26215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:42:47.743844 26215 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:42:47.744760 26215 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:42:47.749704 26215 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cdd06f76e50434d953e5d73046e0b32: Generated new TSK 0
I20260619 18:42:47.750327 26215 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.143.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab.
Entry for principal kudu/127.21.143.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab.
I20260619 18:42:47.777724 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.65:0
--local_ip_for_outbound_sockets=127.21.143.65
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:39435
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:34973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:47.888974 26229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:47.889235 26229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:47.889317 26229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:47.893412 26229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:47.893467 26229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:47.893551 26229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.65
I20260619 18:42:47.898658 26229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:34973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:39435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.65
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:47.899755 26229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:47.900768 26229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:47.909854 26234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:47.909854 26235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:47.909976 26229 server_base.cc:1034] running on GCE node
W20260619 18:42:47.909901 26237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:47.910595 26229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:47.911170 26229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:47.912354 26229 hybrid_clock.cc:648] HybridClock initialized: now 1781894567912307 us; error 58 us; skew 500 ppm
Jun 19 18:42:47 dist-test-slave-mld5 krb5kdc[26123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894567, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:47.918606 26229 init.cc:377] Logged in from keytab as kudu/127.21.143.65@KRBTEST.COM (short username kudu)
I20260619 18:42:47.919554 26229 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:47.919629 26229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:47.919819 26229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:47.922127 26229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-0/data/instance:
uuid: "6376397959664554a2da061962921eef"
format_stamp: "Formatted at 2026-06-19 18:42:47 on dist-test-slave-mld5"
server_key: "8f47e50d0e24151119f285ce73864901"
server_key_iv: "96c7fb05fe12c6dfedf6b8e33980a73d"
server_key_version: "encryptionkey@0"
I20260619 18:42:47.922861 26229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance:
uuid: "6376397959664554a2da061962921eef"
format_stamp: "Formatted at 2026-06-19 18:42:47 on dist-test-slave-mld5"
server_key: "8f47e50d0e24151119f285ce73864901"
server_key_iv: "96c7fb05fe12c6dfedf6b8e33980a73d"
server_key_version: "encryptionkey@0"
I20260619 18:42:47.926944 26229 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 18:42:47.929596 26243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:47.931010 26229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:42:47.931182 26229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-0/wal
uuid: "6376397959664554a2da061962921eef"
format_stamp: "Formatted at 2026-06-19 18:42:47 on dist-test-slave-mld5"
server_key: "8f47e50d0e24151119f285ce73864901"
server_key_iv: "96c7fb05fe12c6dfedf6b8e33980a73d"
server_key_version: "encryptionkey@0"
I20260619 18:42:47.931341 26229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:47.973964 26229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:47.977305 26229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:47.977506 26229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:47.978104 26229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:47.978981 26229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:47.979056 26229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:47.979125 26229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:47.979187 26229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:47.987895 26229 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.65:44083
I20260619 18:42:47.987911 26356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.65:44083 every 8 connection(s)
I20260619 18:42:47.988477 26229 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
I20260619 18:42:47.994881 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 26229
I20260619 18:42:47.995018 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance
I20260619 18:42:47.995339 22077 external_mini_cluster.cc:1428] Setting key a56dcf27240e3f3b33d8afe459ac632b
Jun 19 18:42:47 dist-test-slave-mld5 krb5kdc[26123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894567, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:48.002030 26206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:47.990137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:59385 (local address 127.21.143.126:39435)
0619 18:42:47.990281 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:47.990285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:47.991497 (+ 1212us) server_negotiation.cc:407] Connection header received
0619 18:42:47.992307 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:47.992311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:47.992355 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:47.992429 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:47.993514 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:47.994040 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:47.995139 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:47.995514 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:47.995626 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:47.998405 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:47.998409 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:47.998412 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:47.998436 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:47.999884 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:48.000450 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:48.000457 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:48.000461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:48.000552 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:48.000875 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:48.000881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:48.000886 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:48.001131 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:48.001248 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:48.001766 (+ 518us) server_negotiation.cc:299] Negotiation successful
0619 18:42:48.001901 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 18:42:48.002769 26359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:47.990516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39435 (local address 127.21.143.65:59385)
0619 18:42:47.991343 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:47.991401 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:47.992124 (+ 723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:47.992585 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:47.992592 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:47.992918 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:47.993392 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:47.993404 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:47.994140 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:47.994143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:47.995010 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:47.995023 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:47.995650 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:47.995653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:47.995831 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:47.996487 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:47.996503 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:47.998245 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:48.000055 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:48.000060 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:48.000071 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:48.000346 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:48.000645 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:48.000647 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:48.000648 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:48.000783 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:48.001241 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:48.001246 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:48.001523 (+ 277us) client_negotiation.cc:769] Sending connection context
0619 18:42:48.001678 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 18:42:48.001925 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":126,"threads_started":1}
I20260619 18:42:48.004050 26357 heartbeater.cc:347] Connected to a master server at 127.21.143.126:39435
I20260619 18:42:48.004310 26357 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:48.004833 26357 heartbeater.cc:511] Master 127.21.143.126:39435 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.143.66@KRBTEST.COM; defaulting to no policy
I20260619 18:42:48.006446 26167 ts_manager.cc:194] Registered new tserver with Master: 6376397959664554a2da061962921eef (127.21.143.65:44083)
I20260619 18:42:48.007745 26167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.65@KRBTEST.COM'} at 127.21.143.65:59385
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab.
Entry for principal kudu/127.21.143.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab.
I20260619 18:42:48.022470 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.66:0
--local_ip_for_outbound_sockets=127.21.143.66
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:39435
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:34973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:48.131785 26362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:48.132026 26362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:48.132071 26362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:48.136154 26362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:48.136207 26362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:48.136297 26362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.66
I20260619 18:42:48.141374 26362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:34973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:39435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.66
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:48.142346 26362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:48.143386 26362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:48.152415 26368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:48.152426 26367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:48.152426 26370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:48.152491 26362 server_base.cc:1034] running on GCE node
I20260619 18:42:48.153080 26362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:48.153709 26362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:48.154897 26362 hybrid_clock.cc:648] HybridClock initialized: now 1781894568154884 us; error 29 us; skew 500 ppm
Jun 19 18:42:48 dist-test-slave-mld5 krb5kdc[26123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894568, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:48.157828 26362 init.cc:377] Logged in from keytab as kudu/127.21.143.66@KRBTEST.COM (short username kudu)
I20260619 18:42:48.158602 26362 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:48.158697 26362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:48.158984 26362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:48.161171 26362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-1/data/instance:
uuid: "fb69c5b3ff2c4e0b980a653dcdc3ed2e"
format_stamp: "Formatted at 2026-06-19 18:42:48 on dist-test-slave-mld5"
server_key: "f25572d716ba3e8fc6d15dcfdbc5f909"
server_key_iv: "a4d55015af109a74e3d40c61fef17b1c"
server_key_version: "encryptionkey@0"
I20260619 18:42:48.161734 26362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance:
uuid: "fb69c5b3ff2c4e0b980a653dcdc3ed2e"
format_stamp: "Formatted at 2026-06-19 18:42:48 on dist-test-slave-mld5"
server_key: "f25572d716ba3e8fc6d15dcfdbc5f909"
server_key_iv: "a4d55015af109a74e3d40c61fef17b1c"
server_key_version: "encryptionkey@0"
I20260619 18:42:48.165526 26362 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:42:48.168164 26376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:48.169596 26362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:42:48.169770 26362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-1/wal
uuid: "fb69c5b3ff2c4e0b980a653dcdc3ed2e"
format_stamp: "Formatted at 2026-06-19 18:42:48 on dist-test-slave-mld5"
server_key: "f25572d716ba3e8fc6d15dcfdbc5f909"
server_key_iv: "a4d55015af109a74e3d40c61fef17b1c"
server_key_version: "encryptionkey@0"
I20260619 18:42:48.169893 26362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:48.198683 26362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:48.202108 26362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:48.202313 26362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:48.202903 26362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:48.203794 26362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:48.203866 26362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:48.203936 26362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:48.203980 26362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:48.212715 26362 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.66:33433
I20260619 18:42:48.212726 26489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.66:33433 every 8 connection(s)
I20260619 18:42:48.213282 26362 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
I20260619 18:42:48.219105 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 26362
I20260619 18:42:48.219236 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance
I20260619 18:42:48.219565 22077 external_mini_cluster.cc:1428] Setting key d87f58fd3c9014a5ecfb77e5f1efd323
Jun 19 18:42:48 dist-test-slave-mld5 krb5kdc[26123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894568, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:48.228971 26206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:48.215026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:45511 (local address 127.21.143.126:39435)
0619 18:42:48.215170 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:48.215174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:48.216046 (+ 872us) server_negotiation.cc:407] Connection header received
0619 18:42:48.217344 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:48.217347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:48.217403 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:48.217502 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:48.218809 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:48.219694 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:48.221287 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:48.221648 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:48.221759 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:48.224962 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:48.224969 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:48.224975 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:48.225007 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:48.226758 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:48.227247 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:48.227254 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:48.227258 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:48.227375 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:48.227696 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:48.227702 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:48.227706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:48.227957 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:48.228066 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:48.228630 (+ 564us) server_negotiation.cc:299] Negotiation successful
0619 18:42:48.228774 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 18:42:48.229714 26492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:48.215444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39435 (local address 127.21.143.66:45511)
0619 18:42:48.215896 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:48.215935 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:48.217127 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:48.217668 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:48.217680 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:48.218171 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:48.218643 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:48.218656 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:48.219824 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:48.219826 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:48.221132 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:48.221142 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:48.221842 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:48.221848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:48.222086 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:48.222753 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:48.223136 (+ 383us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:48.224798 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:48.226885 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:48.226890 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:48.226901 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:48.227140 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:48.227471 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:48.227474 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:48.227475 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:48.227601 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:48.228083 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:48.228088 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:48.228371 (+ 283us) client_negotiation.cc:769] Sending connection context
0619 18:42:48.228546 (+ 175us) client_negotiation.cc:240] Negotiation successful
0619 18:42:48.228795 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for kudu/127.21.143.67@KRBTEST.COM; defaulting to no policy
I20260619 18:42:48.230933 26490 heartbeater.cc:347] Connected to a master server at 127.21.143.126:39435
I20260619 18:42:48.231141 26490 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:48.231621 26490 heartbeater.cc:511] Master 127.21.143.126:39435 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.67@KRBTEST.COM" created.
I20260619 18:42:48.232762 26167 ts_manager.cc:194] Registered new tserver with Master: fb69c5b3ff2c4e0b980a653dcdc3ed2e (127.21.143.66:33433)
I20260619 18:42:48.233355 26167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.66@KRBTEST.COM'} at 127.21.143.66:45511
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab.
Entry for principal kudu/127.21.143.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab.
I20260619 18:42:48.247092 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.67:0
--local_ip_for_outbound_sockets=127.21.143.67
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:39435
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:34973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:48.357964 26495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:48.358212 26495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:48.358296 26495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:48.362434 26495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:48.362496 26495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:48.362556 26495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.67
I20260619 18:42:48.368461 26495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:34973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:39435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.67
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:48.369488 26495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:48.370471 26495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:48.379989 26500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:48.380010 26503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:48.379992 26501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:48.380352 26495 server_base.cc:1034] running on GCE node
I20260619 18:42:48.380769 26495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:48.381435 26495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:48.382613 26495 hybrid_clock.cc:648] HybridClock initialized: now 1781894568382576 us; error 47 us; skew 500 ppm
Jun 19 18:42:48 dist-test-slave-mld5 krb5kdc[26123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894568, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:48.385357 26495 init.cc:377] Logged in from keytab as kudu/127.21.143.67@KRBTEST.COM (short username kudu)
I20260619 18:42:48.386063 26495 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:48.386155 26495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:48.386394 26495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:48.388448 26495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-2/data/instance:
uuid: "a55ff57b47ef4a9786f366d5e125394d"
format_stamp: "Formatted at 2026-06-19 18:42:48 on dist-test-slave-mld5"
server_key: "584b83f39e46cea0c275977dac594a97"
server_key_iv: "6404b274e07da484b109f79bf2ae0948"
server_key_version: "encryptionkey@0"
I20260619 18:42:48.389001 26495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance:
uuid: "a55ff57b47ef4a9786f366d5e125394d"
format_stamp: "Formatted at 2026-06-19 18:42:48 on dist-test-slave-mld5"
server_key: "584b83f39e46cea0c275977dac594a97"
server_key_iv: "6404b274e07da484b109f79bf2ae0948"
server_key_version: "encryptionkey@0"
I20260619 18:42:48.392786 26495 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:42:48.395171 26509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:48.396541 26495 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:42:48.396699 26495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-2/wal
uuid: "a55ff57b47ef4a9786f366d5e125394d"
format_stamp: "Formatted at 2026-06-19 18:42:48 on dist-test-slave-mld5"
server_key: "584b83f39e46cea0c275977dac594a97"
server_key_iv: "6404b274e07da484b109f79bf2ae0948"
server_key_version: "encryptionkey@0"
I20260619 18:42:48.396824 26495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:48.416966 26495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:48.419912 26495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:48.420110 26495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:48.420702 26495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:48.421566 26495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:48.421629 26495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:48.421698 26495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:48.421772 26495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:48.430693 26495 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.67:43031
I20260619 18:42:48.430713 26622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.67:43031 every 8 connection(s)
I20260619 18:42:48.431278 26495 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
I20260619 18:42:48.433333 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 26495
I20260619 18:42:48.433467 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance
I20260619 18:42:48.433827 22077 external_mini_cluster.cc:1428] Setting key 7261a9d9b46ce48ae85fbd57867360bd
Jun 19 18:42:48 dist-test-slave-mld5 krb5kdc[26123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894568, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:48.445369 26206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:48.433006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:60943 (local address 127.21.143.126:39435)
0619 18:42:48.433138 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:48.433145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:48.434353 (+ 1208us) server_negotiation.cc:407] Connection header received
0619 18:42:48.436275 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:48.436277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:48.436323 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:48.436392 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:48.437571 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:48.438118 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:48.439228 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:48.439535 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:48.439616 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:48.441971 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:48.441974 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:48.441976 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:48.442001 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:48.443553 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:48.444017 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:48.444019 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:48.444021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:48.444096 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:48.444420 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:48.444423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:48.444424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:48.444619 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:48.444702 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:48.445143 (+ 441us) server_negotiation.cc:299] Negotiation successful
0619 18:42:48.445245 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 18:42:48.446323 26625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:48.433526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39435 (local address 127.21.143.67:60943)
0619 18:42:48.434187 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:48.434246 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:48.436068 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:48.436556 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:48.436562 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:48.436925 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:48.437423 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:48.437436 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:48.438224 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:48.438226 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:48.439110 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:48.439118 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:48.439663 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:48.439666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:48.439819 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:48.440375 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:48.440391 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:48.441816 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:48.443645 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:48.443650 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:48.443684 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:48.443929 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:48.444172 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:48.444175 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:48.444176 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:48.444306 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:48.444707 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:48.444712 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:48.444942 (+ 230us) client_negotiation.cc:769] Sending connection context
0619 18:42:48.445096 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 18:42:48.445351 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":138,"threads_started":1}
I20260619 18:42:48.447605 26623 heartbeater.cc:347] Connected to a master server at 127.21.143.126:39435
I20260619 18:42:48.447857 26623 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:48.448261 26623 heartbeater.cc:511] Master 127.21.143.126:39435 requested a full tablet report, sending...
I20260619 18:42:48.449169 26167 ts_manager.cc:194] Registered new tserver with Master: a55ff57b47ef4a9786f366d5e125394d (127.21.143.67:43031)
I20260619 18:42:48.449679 26167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.67@KRBTEST.COM'} at 127.21.143.67:60943
I20260619 18:42:48.459865 22077 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:42:48.469374 26206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:48.461772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40944 (local address 127.21.143.126:39435)
0619 18:42:48.461913 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:48.461917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:48.462192 (+ 275us) server_negotiation.cc:407] Connection header received
0619 18:42:48.462344 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:48.462348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:48.462396 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:48.462490 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:48.463180 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:48.463791 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:48.464709 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:48.464952 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:48.465062 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:48.466336 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:48.466340 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:48.466343 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:48.466370 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:48.467676 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:48.468101 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:48.468104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:48.468106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:48.468188 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:48.468478 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:48.468481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:48.468482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:48.468706 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:48.468846 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:48.469085 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 18:42:48.469208 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 18:42:48.472055 26167 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:42:48.475227 26167 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:42:48.489473 26638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:48.484456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:43031 (local address 127.0.0.1:46226)
0619 18:42:48.485107 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:48.485119 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:48.485194 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:48.485604 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:48.485607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:48.485623 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:48.485722 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:48.485729 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:48.486709 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:48.486713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:48.487947 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:48.487957 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:48.488946 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:48.488949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:48.489057 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:48.489092 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 18:42:48.489197 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 18:42:48.489315 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":3,"thread_start_us":72,"threads_started":1}
I20260619 18:42:48.489477 26639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:48.484827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46226 (local address 127.21.143.67:43031)
0619 18:42:48.485203 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:48.485216 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:48.485241 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:42:48.485325 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:48.485329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:48.485471 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:48.485634 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:48.485815 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:48.486564 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:48.488071 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:48.488823 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:48.488948 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:48.489100 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:48.489235 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 18:42:48.489327 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":168,"threads_started":1}
I20260619 18:42:48.489873 26634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:48.483676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:44083 (local address 127.0.0.1:37958)
0619 18:42:48.484412 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:48.484466 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:48.484588 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:48.485294 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:48.485302 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:48.485329 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:48.485485 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:48.485492 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:48.487103 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:48.487106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:48.488282 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:48.488289 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:48.489346 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:48.489352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:48.489559 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:48.489587 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 18:42:48.489647 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 18:42:48.489728 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":143,"thread_start_us":100,"threads_started":1}
I20260619 18:42:48.489912 26636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:48.484143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:33433 (local address 127.0.0.1:44744)
0619 18:42:48.484821 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:48.484833 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:48.484909 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:48.485759 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:48.485762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:48.485780 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:48.485858 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:48.485863 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:48.487347 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:48.487351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:48.488441 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:48.488451 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:48.489439 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:48.489443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:48.489523 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:48.489697 (+ 174us) client_negotiation.cc:769] Sending connection context
0619 18:42:48.489752 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 18:42:48.489805 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":1,"thread_start_us":113,"threads_started":1}
I20260619 18:42:48.490620 26637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:48.484314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44744 (local address 127.21.143.66:33433)
0619 18:42:48.485307 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:48.485318 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:48.485338 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:42:48.485409 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:48.485414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:48.485616 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:48.486009 (+ 393us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:48.486157 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:48.487190 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:48.488561 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:48.489327 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:48.490208 (+ 881us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:48.490342 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:48.490390 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 18:42:48.490482 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":890,"thread_start_us":142,"threads_started":1}
I20260619 18:42:48.490626 26635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:48.483886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37958 (local address 127.21.143.65:44083)
0619 18:42:48.484745 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:48.484753 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:48.484780 (+ 27us) server_negotiation.cc:407] Connection header received
0619 18:42:48.484860 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:48.484868 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:48.485095 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:48.485637 (+ 542us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:48.485809 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:48.486961 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:48.488390 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:48.489177 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:48.490156 (+ 979us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:48.490324 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:48.490374 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 18:42:48.490473 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":111,"threads_started":1}
I20260619 18:42:48.492550 26557 tablet_service.cc:1467] Processing CreateTablet for tablet bbf74c579ff641cd934d50c6a91aa485 (DEFAULT_TABLE table=test-table [id=7b4634a5b3d24503b393952fc555d0bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:42:48.492926 26424 tablet_service.cc:1467] Processing CreateTablet for tablet bbf74c579ff641cd934d50c6a91aa485 (DEFAULT_TABLE table=test-table [id=7b4634a5b3d24503b393952fc555d0bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:42:48.493631 26557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbf74c579ff641cd934d50c6a91aa485. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:48.493924 26424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbf74c579ff641cd934d50c6a91aa485. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:48.494805 26291 tablet_service.cc:1467] Processing CreateTablet for tablet bbf74c579ff641cd934d50c6a91aa485 (DEFAULT_TABLE table=test-table [id=7b4634a5b3d24503b393952fc555d0bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:42:48.495795 26291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbf74c579ff641cd934d50c6a91aa485. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:48.499976 26640 tablet_bootstrap.cc:492] T bbf74c579ff641cd934d50c6a91aa485 P fb69c5b3ff2c4e0b980a653dcdc3ed2e: Bootstrap starting.
I20260619 18:42:48.501502 26641 tablet_bootstrap.cc:492] T bbf74c579ff641cd934d50c6a91aa485 P a55ff57b47ef4a9786f366d5e125394d: Bootstrap starting.
I20260619 18:42:48.502306 26640 tablet_bootstrap.cc:654] T bbf74c579ff641cd934d50c6a91aa485 P fb69c5b3ff2c4e0b980a653dcdc3ed2e: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:48.503036 26642 tablet_bootstrap.cc:492] T bbf74c579ff641cd934d50c6a91aa485 P 6376397959664554a2da061962921eef: Bootstrap starting.
I20260619 18:42:48.503683 26641 tablet_bootstrap.cc:654] T bbf74c579ff641cd934d50c6a91aa485 P a55ff57b47ef4a9786f366d5e125394d: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:48.503854 26640 log.cc:824] T bbf74c579ff641cd934d50c6a91aa485 P fb69c5b3ff2c4e0b980a653dcdc3ed2e: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:48.504964 26641 log.cc:824] T bbf74c579ff641cd934d50c6a91aa485 P a55ff57b47ef4a9786f366d5e125394d: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:48.505282 26642 tablet_bootstrap.cc:654] T bbf74c579ff641cd934d50c6a91aa485 P 6376397959664554a2da061962921eef: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:48.506064 26642 log.cc:824] T bbf74c579ff641cd934d50c6a91aa485 P 6376397959664554a2da061962921eef: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:48.506557 26640 tablet_bootstrap.cc:492] T bbf74c579ff641cd934d50c6a91aa485 P fb69c5b3ff2c4e0b980a653dcdc3ed2e: No bootstrap required, opened a new log
I20260619 18:42:48.506817 26641 tablet_bootstrap.cc:492] T bbf74c579ff641cd934d50c6a91aa485 P a55ff57b47ef4a9786f366d5e125394d: No bootstrap required, opened a new log
I20260619 18:42:48.507012 26641 ts_tablet_manager.cc:1397] T bbf74c579ff641cd934d50c6a91aa485 P a55ff57b47ef4a9786f366d5e125394d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 18:42:48.507010 26640 ts_tablet_manager.cc:1397] T bbf74c579ff641cd934d50c6a91aa485 P fb69c5b3ff2c4e0b980a653dcdc3ed2e: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 18:42:48.507934 26642 tablet_bootstrap.cc:492] T bbf74c579ff641cd934d50c6a91aa485 P 6376397959664554a2da061962921eef: No bootstrap required, opened a new log
I20260619 18:42:48.508108 26642 ts_tablet_manager.cc:1397] T bbf74c579ff641cd934d50c6a91aa485 P 6376397959664554a2da061962921eef: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 18:42:48.510098 26641 raft_consensus.cc:348] T bbf74c579ff641cd934d50c6a91aa485 P a55ff57b47ef4a9786f366d5e125394d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6376397959664554a2da061962921eef" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44083 } } peers { permanent_uuid: "a55ff57b47ef4a9786f366d5e125394d" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43031 } } peers { permanent_uuid: "fb69c5b3ff2c4e0b980a653dcdc3ed2e" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33433 } }
I20260619 18:42:48.510357 26641 raft_consensus.cc:374] T bbf74c579ff641cd934d50c6a91aa485 P a55ff57b47ef4a9786f366d5e125394d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:48.510432 26641 raft_consensus.cc:729] T bbf74c579ff641cd934d50c6a91aa485 P a55ff57b47ef4a9786f366d5e125394d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a55ff57b47ef4a9786f366d5e125394d, State: Initialized, Role: FOLLOWER
I20260619 18:42:48.510900 26641 consensus_queue.cc:260] T bbf74c579ff641cd934d50c6a91aa485 P a55ff57b47ef4a9786f366d5e125394d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6376397959664554a2da061962921eef" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44083 } } peers { permanent_uuid: "a55ff57b47ef4a9786f366d5e125394d" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43031 } } peers { permanent_uuid: "fb69c5b3ff2c4e0b980a653dcdc3ed2e" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33433 } }
I20260619 18:42:48.511003 26640 raft_consensus.cc:348] T bbf74c579ff641cd934d50c6a91aa485 P fb69c5b3ff2c4e0b980a653dcdc3ed2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6376397959664554a2da061962921eef" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44083 } } peers { permanent_uuid: "a55ff57b47ef4a9786f366d5e125394d" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43031 } } peers { permanent_uuid: "fb69c5b3ff2c4e0b980a653dcdc3ed2e" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33433 } }
I20260619 18:42:48.511618 26623 heartbeater.cc:503] Master 127.21.143.126:39435 was elected leader, sending a full tablet report...
I20260619 18:42:48.511945 26641 ts_tablet_manager.cc:1428] T bbf74c579ff641cd934d50c6a91aa485 P a55ff57b47ef4a9786f366d5e125394d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 18:42:48.511287 26640 raft_consensus.cc:374] T bbf74c579ff641cd934d50c6a91aa485 P fb69c5b3ff2c4e0b980a653dcdc3ed2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:48.512373 26640 raft_consensus.cc:729] T bbf74c579ff641cd934d50c6a91aa485 P fb69c5b3ff2c4e0b980a653dcdc3ed2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb69c5b3ff2c4e0b980a653dcdc3ed2e, State: Initialized, Role: FOLLOWER
I20260619 18:42:48.512352 26642 raft_consensus.cc:348] T bbf74c579ff641cd934d50c6a91aa485 P 6376397959664554a2da061962921eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6376397959664554a2da061962921eef" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44083 } } peers { permanent_uuid: "a55ff57b47ef4a9786f366d5e125394d" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43031 } } peers { permanent_uuid: "fb69c5b3ff2c4e0b980a653dcdc3ed2e" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33433 } }
I20260619 18:42:48.512593 26642 raft_consensus.cc:374] T bbf74c579ff641cd934d50c6a91aa485 P 6376397959664554a2da061962921eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:48.512830 26642 raft_consensus.cc:729] T bbf74c579ff641cd934d50c6a91aa485 P 6376397959664554a2da061962921eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6376397959664554a2da061962921eef, State: Initialized, Role: FOLLOWER
I20260619 18:42:48.512813 26640 consensus_queue.cc:260] T bbf74c579ff641cd934d50c6a91aa485 P fb69c5b3ff2c4e0b980a653dcdc3ed2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6376397959664554a2da061962921eef" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44083 } } peers { permanent_uuid: "a55ff57b47ef4a9786f366d5e125394d" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43031 } } peers { permanent_uuid: "fb69c5b3ff2c4e0b980a653dcdc3ed2e" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33433 } }
I20260619 18:42:48.513218 26642 consensus_queue.cc:260] T bbf74c579ff641cd934d50c6a91aa485 P 6376397959664554a2da061962921eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6376397959664554a2da061962921eef" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44083 } } peers { permanent_uuid: "a55ff57b47ef4a9786f366d5e125394d" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43031 } } peers { permanent_uuid: "fb69c5b3ff2c4e0b980a653dcdc3ed2e" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33433 } }
I20260619 18:42:48.513859 26357 heartbeater.cc:503] Master 127.21.143.126:39435 was elected leader, sending a full tablet report...
I20260619 18:42:48.514297 26642 ts_tablet_manager.cc:1428] T bbf74c579ff641cd934d50c6a91aa485 P 6376397959664554a2da061962921eef: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 18:42:48.515690 26640 ts_tablet_manager.cc:1428] T bbf74c579ff641cd934d50c6a91aa485 P fb69c5b3ff2c4e0b980a653dcdc3ed2e: Time spent starting tablet: real 0.009s user 0.002s sys 0.004s
I20260619 18:42:48.516077 26490 heartbeater.cc:503] Master 127.21.143.126:39435 was elected leader, sending a full tablet report...
I20260619 18:42:48.607483 26648 raft_consensus.cc:482] T bbf74c579ff641cd934d50c6a91aa485 P fb69c5b3ff2c4e0b980a653dcdc3ed2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:42:48.607707 26648 raft_consensus.cc:504] T bbf74c579ff641cd934d50c6a91aa485 P fb69c5b3ff2c4e0b980a653dcdc3ed2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6376397959664554a2da061962921eef" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44083 } } peers { permanent_uuid: "a55ff57b47ef4a9786f366d5e125394d" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43031 } } peers { permanent_uuid: "fb69c5b3ff2c4e0b980a653dcdc3ed2e" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33433 } }
I20260619 18:42:48.608897 26648 leader_election.cc:288] T bbf74c579ff641cd934d50c6a91aa485 P fb69c5b3ff2c4e0b980a653dcdc3ed2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6376397959664554a2da061962921eef (127.21.143.65:44083), a55ff57b47ef4a9786f366d5e125394d (127.21.143.67:43031)
I20260619 18:42:48.612962 26635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:48.609415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:37053 (local address 127.21.143.65:44083)
0619 18:42:48.609570 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:48.609574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:48.609595 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:42:48.609661 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:48.609667 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:48.609718 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:48.609841 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:48.610221 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:48.610846 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:48.611953 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:48.612392 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:48.612509 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:48.612570 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:48.612770 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 18:42:48.612834 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 18:42:48.612973 26639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:48.609649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:56761 (local address 127.21.143.67:43031)
0619 18:42:48.609785 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:48.609789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:48.609958 (+ 169us) server_negotiation.cc:407] Connection header received
0619 18:42:48.610073 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:48.610076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:48.610129 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:48.610203 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:48.610468 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:48.610970 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:48.612042 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:48.612415 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:48.612547 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:48.612600 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:48.612801 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 18:42:48.612867 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 18:42:48.612964 26492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:48.609181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:44083 (local address 127.21.143.66:37053)
0619 18:42:48.609312 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:48.609330 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:48.609431 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:48.609875 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:48.609877 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:48.609896 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:48.610076 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:48.610083 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:48.610954 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:48.610957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:48.611822 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:48.611831 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:48.612509 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:48.612513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:48.612628 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:48.612649 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 18:42:48.612720 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 18:42:48.612809 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":1,"spinlock_wait_cycles":20352}
I20260619 18:42:48.612972 26649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:48.609491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:43031 (local address 127.21.143.66:56761)
0619 18:42:48.609853 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:48.609867 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:48.609979 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:48.610234 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:48.610237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:48.610254 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:48.610365 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:48.610371 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:48.611074 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:48.611076 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:48.611923 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:48.611936 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:48.612561 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:48.612564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:48.612663 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:48.612679 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 18:42:48.612774 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 18:42:48.612836 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":126,"threads_started":1}
I20260619 18:42:48.613641 26311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbf74c579ff641cd934d50c6a91aa485" candidate_uuid: "fb69c5b3ff2c4e0b980a653dcdc3ed2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6376397959664554a2da061962921eef" is_pre_election: true
I20260619 18:42:48.613638 26577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbf74c579ff641cd934d50c6a91aa485" candidate_uuid: "fb69c5b3ff2c4e0b980a653dcdc3ed2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a55ff57b47ef4a9786f366d5e125394d" is_pre_election: true
I20260619 18:42:48.613936 26311 raft_consensus.cc:2454] T bbf74c579ff641cd934d50c6a91aa485 P 6376397959664554a2da061962921eef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb69c5b3ff2c4e0b980a653dcdc3ed2e in term 0.
I20260619 18:42:48.613968 26577 raft_consensus.cc:2454] T bbf74c579ff641cd934d50c6a91aa485 P a55ff57b47ef4a9786f366d5e125394d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb69c5b3ff2c4e0b980a653dcdc3ed2e in term 0.
I20260619 18:42:48.614529 26377 leader_election.cc:302] T bbf74c579ff641cd934d50c6a91aa485 P fb69c5b3ff2c4e0b980a653dcdc3ed2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6376397959664554a2da061962921eef, fb69c5b3ff2c4e0b980a653dcdc3ed2e; no voters:
I20260619 18:42:48.614845 26648 raft_consensus.cc:2781] T bbf74c579ff641cd934d50c6a91aa485 P fb69c5b3ff2c4e0b980a653dcdc3ed2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:42:48.614964 26648 raft_consensus.cc:482] T bbf74c579ff641cd934d50c6a91aa485 P fb69c5b3ff2c4e0b980a653dcdc3ed2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:42:48.615021 26648 raft_consensus.cc:3037] T bbf74c579ff641cd934d50c6a91aa485 P fb69c5b3ff2c4e0b980a653dcdc3ed2e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:48.616118 26648 raft_consensus.cc:504] T bbf74c579ff641cd934d50c6a91aa485 P fb69c5b3ff2c4e0b980a653dcdc3ed2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6376397959664554a2da061962921eef" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44083 } } peers { permanent_uuid: "a55ff57b47ef4a9786f366d5e125394d" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43031 } } peers { permanent_uuid: "fb69c5b3ff2c4e0b980a653dcdc3ed2e" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33433 } }
I20260619 18:42:48.616446 26648 leader_election.cc:288] T bbf74c579ff641cd934d50c6a91aa485 P fb69c5b3ff2c4e0b980a653dcdc3ed2e [CANDIDATE]: Term 1 election: Requested vote from peers 6376397959664554a2da061962921eef (127.21.143.65:44083), a55ff57b47ef4a9786f366d5e125394d (127.21.143.67:43031)
I20260619 18:42:48.616830 26311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbf74c579ff641cd934d50c6a91aa485" candidate_uuid: "fb69c5b3ff2c4e0b980a653dcdc3ed2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6376397959664554a2da061962921eef"
I20260619 18:42:48.616896 26577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbf74c579ff641cd934d50c6a91aa485" candidate_uuid: "fb69c5b3ff2c4e0b980a653dcdc3ed2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a55ff57b47ef4a9786f366d5e125394d"
I20260619 18:42:48.616976 26311 raft_consensus.cc:3037] T bbf74c579ff641cd934d50c6a91aa485 P 6376397959664554a2da061962921eef [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:48.617033 26577 raft_consensus.cc:3037] T bbf74c579ff641cd934d50c6a91aa485 P a55ff57b47ef4a9786f366d5e125394d [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:48.617971 26311 raft_consensus.cc:2454] T bbf74c579ff641cd934d50c6a91aa485 P 6376397959664554a2da061962921eef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb69c5b3ff2c4e0b980a653dcdc3ed2e in term 1.
I20260619 18:42:48.618101 26577 raft_consensus.cc:2454] T bbf74c579ff641cd934d50c6a91aa485 P a55ff57b47ef4a9786f366d5e125394d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb69c5b3ff2c4e0b980a653dcdc3ed2e in term 1.
I20260619 18:42:48.618276 26377 leader_election.cc:302] T bbf74c579ff641cd934d50c6a91aa485 P fb69c5b3ff2c4e0b980a653dcdc3ed2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6376397959664554a2da061962921eef, fb69c5b3ff2c4e0b980a653dcdc3ed2e; no voters:
I20260619 18:42:48.618472 26648 raft_consensus.cc:2781] T bbf74c579ff641cd934d50c6a91aa485 P fb69c5b3ff2c4e0b980a653dcdc3ed2e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:48.618711 26648 raft_consensus.cc:686] T bbf74c579ff641cd934d50c6a91aa485 P fb69c5b3ff2c4e0b980a653dcdc3ed2e [term 1 LEADER]: Becoming Leader. State: Replica: fb69c5b3ff2c4e0b980a653dcdc3ed2e, State: Running, Role: LEADER
I20260619 18:42:48.619057 26648 consensus_queue.cc:237] T bbf74c579ff641cd934d50c6a91aa485 P fb69c5b3ff2c4e0b980a653dcdc3ed2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6376397959664554a2da061962921eef" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44083 } } peers { permanent_uuid: "a55ff57b47ef4a9786f366d5e125394d" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43031 } } peers { permanent_uuid: "fb69c5b3ff2c4e0b980a653dcdc3ed2e" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33433 } }
I20260619 18:42:48.622459 26167 catalog_manager.cc:5495] T bbf74c579ff641cd934d50c6a91aa485 P fb69c5b3ff2c4e0b980a653dcdc3ed2e reported cstate change: term changed from 0 to 1, leader changed from <none> to fb69c5b3ff2c4e0b980a653dcdc3ed2e (127.21.143.66). New cstate: current_term: 1 leader_uuid: "fb69c5b3ff2c4e0b980a653dcdc3ed2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6376397959664554a2da061962921eef" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a55ff57b47ef4a9786f366d5e125394d" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb69c5b3ff2c4e0b980a653dcdc3ed2e" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 33433 } health_report { overall_health: HEALTHY } } }
I20260619 18:42:48.640795 26637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:48.637151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44748 (local address 127.21.143.66:33433)
0619 18:42:48.637279 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:48.637283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:48.637298 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:42:48.637476 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:48.637479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:48.637534 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:48.637638 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:42:48.637898 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:48.638411 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:48.639252 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:48.639547 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:48.639656 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:48.639974 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:42:48.640400 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:42:48.640531 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:48.640592 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 18:42:48.640647 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 18:42:48.646755 26577 raft_consensus.cc:1261] T bbf74c579ff641cd934d50c6a91aa485 P a55ff57b47ef4a9786f366d5e125394d [term 1 FOLLOWER]: Refusing update from remote peer fb69c5b3ff2c4e0b980a653dcdc3ed2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:42:48.646755 26311 raft_consensus.cc:1261] T bbf74c579ff641cd934d50c6a91aa485 P 6376397959664554a2da061962921eef [term 1 FOLLOWER]: Refusing update from remote peer fb69c5b3ff2c4e0b980a653dcdc3ed2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:42:48.647531 26648 consensus_queue.cc:1035] T bbf74c579ff641cd934d50c6a91aa485 P fb69c5b3ff2c4e0b980a653dcdc3ed2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6376397959664554a2da061962921eef" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:42:48.647773 26650 consensus_queue.cc:1035] T bbf74c579ff641cd934d50c6a91aa485 P fb69c5b3ff2c4e0b980a653dcdc3ed2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a55ff57b47ef4a9786f366d5e125394d" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 43031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:42:48.654605 26653 mvcc.cc:205] Tried to move back new op lower bound from 7298640153169309696 to 7298640153066430464. Current Snapshot: MvccSnapshot[applied={T|T < 7298640153169309696}]
I20260619 18:42:48.657037 26654 mvcc.cc:205] Tried to move back new op lower bound from 7298640153169309696 to 7298640153066430464. Current Snapshot: MvccSnapshot[applied={T|T < 7298640153169309696}]
I20260619 18:42:48.663959 26167 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:42:48.664136 26167 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:42:48.666476 26167 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3cdd06f76e50434d953e5d73046e0b32: Sending DeleteTablet for 3 replicas of tablet bbf74c579ff641cd934d50c6a91aa485
I20260619 18:42:48.667382 26291 tablet_service.cc:1514] Processing DeleteTablet for tablet bbf74c579ff641cd934d50c6a91aa485 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:42:48 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:37958
I20260619 18:42:48.667482 26557 tablet_service.cc:1514] Processing DeleteTablet for tablet bbf74c579ff641cd934d50c6a91aa485 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:42:48 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:46226
I20260619 18:42:48.667896 26424 tablet_service.cc:1514] Processing DeleteTablet for tablet bbf74c579ff641cd934d50c6a91aa485 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:42:48 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:44744
I20260619 18:42:48.668093 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 26229
I20260619 18:42:48.668108 26664 tablet_replica.cc:330] T bbf74c579ff641cd934d50c6a91aa485 P a55ff57b47ef4a9786f366d5e125394d: stopping tablet replica
I20260619 18:42:48.668413 26664 raft_consensus.cc:2229] T bbf74c579ff641cd934d50c6a91aa485 P a55ff57b47ef4a9786f366d5e125394d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:42:48.668702 26664 raft_consensus.cc:2258] T bbf74c579ff641cd934d50c6a91aa485 P a55ff57b47ef4a9786f366d5e125394d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:42:48.670228 26664 ts_tablet_manager.cc:1905] T bbf74c579ff641cd934d50c6a91aa485 P a55ff57b47ef4a9786f366d5e125394d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:42:48.672890 26664 ts_tablet_manager.cc:1918] T bbf74c579ff641cd934d50c6a91aa485 P a55ff57b47ef4a9786f366d5e125394d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:42:48.673015 26664 log.cc:1196] T bbf74c579ff641cd934d50c6a91aa485 P a55ff57b47ef4a9786f366d5e125394d: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894542572984-22077-0/minicluster-data/ts-2/wal/wals/bbf74c579ff641cd934d50c6a91aa485
I20260619 18:42:48.673372 26664 ts_tablet_manager.cc:1939] T bbf74c579ff641cd934d50c6a91aa485 P a55ff57b47ef4a9786f366d5e125394d: Deleting consensus metadata
I20260619 18:42:48.674453 26151 catalog_manager.cc:4841] TS a55ff57b47ef4a9786f366d5e125394d (127.21.143.67:43031): tablet bbf74c579ff641cd934d50c6a91aa485 (table test-table [id=7b4634a5b3d24503b393952fc555d0bc]) successfully deleted
I20260619 18:42:48.675485 26665 tablet_replica.cc:330] T bbf74c579ff641cd934d50c6a91aa485 P fb69c5b3ff2c4e0b980a653dcdc3ed2e: stopping tablet replica
I20260619 18:42:48.675770 26665 raft_consensus.cc:2229] T bbf74c579ff641cd934d50c6a91aa485 P fb69c5b3ff2c4e0b980a653dcdc3ed2e [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:42:48.676112 26665 raft_consensus.cc:2258] T bbf74c579ff641cd934d50c6a91aa485 P fb69c5b3ff2c4e0b980a653dcdc3ed2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:42:48.677407 26665 ts_tablet_manager.cc:1905] T bbf74c579ff641cd934d50c6a91aa485 P fb69c5b3ff2c4e0b980a653dcdc3ed2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:42:48.678108 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 26362
W20260619 18:42:48.678079 26151 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.143.65:44083 (error 108)
I20260619 18:42:48.679507 26636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:48.679150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:44083 (local address 127.0.0.1:37964)
0619 18:42:48.679328 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:48.679409 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.65:44083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 18:42:48.680030 26151 catalog_manager.cc:4584] TS 6376397959664554a2da061962921eef (127.21.143.65:44083): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bbf74c579ff641cd934d50c6a91aa485: Network error: Client connection negotiation failed: client connection to 127.21.143.65:44083: connect: Connection refused (error 111)
I20260619 18:42:48.683972 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 26495
I20260619 18:42:48.684873 26636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:48.684123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:33433 (local address 127.0.0.1:44760)
0619 18:42:48.684708 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:48.684775 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.66:33433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481}
I20260619 18:42:48.689574 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 26136
2026-06-19T18:42:48Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3996 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T18:42:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:42:48Z Disabled control of system clock
I20260619 18:42:48.719473 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:34805
--webserver_interface=127.21.143.126
--webserver_port=0
--builtin_ntp_servers=127.21.143.84:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:34805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 18:42:48.830907 26670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:48.831194 26670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:48.831233 26670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:48.835372 26670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:42:48.835431 26670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:48.835446 26670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:48.835464 26670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:42:48.835479 26670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:42:48.840241 26670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:34805
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:34805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:48.841293 26670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:48.842684 26670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:48.851480 26675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:48.851491 26676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:48.851608 26670 server_base.cc:1034] running on GCE node
W20260619 18:42:48.851498 26678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:48.852303 26670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:48.853432 26670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:48.854622 26670 hybrid_clock.cc:648] HybridClock initialized: now 1781894568854607 us; error 26 us; skew 500 ppm
I20260619 18:42:48.856500 26670 webserver.cc:460] Webserver started at http://127.21.143.126:33421/ using document root <none> and password file <none>
I20260619 18:42:48.857084 26670 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:48.857174 26670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:48.857396 26670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:48.859520 26670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "75d768e855db4ac2870dfc64a2ec7ac5"
format_stamp: "Formatted at 2026-06-19 18:42:48 on dist-test-slave-mld5"
server_key: "5dba1a20a364f11ae831286e3fd98d36"
server_key_iv: "53578ec9922ec0d377a7fc110a3cce16"
server_key_version: "encryptionkey@0"
I20260619 18:42:48.860054 26670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "75d768e855db4ac2870dfc64a2ec7ac5"
format_stamp: "Formatted at 2026-06-19 18:42:48 on dist-test-slave-mld5"
server_key: "5dba1a20a364f11ae831286e3fd98d36"
server_key_iv: "53578ec9922ec0d377a7fc110a3cce16"
server_key_version: "encryptionkey@0"
I20260619 18:42:48.863620 26670 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:42:48.865886 26684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:48.867041 26670 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:42:48.867172 26670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "75d768e855db4ac2870dfc64a2ec7ac5"
format_stamp: "Formatted at 2026-06-19 18:42:48 on dist-test-slave-mld5"
server_key: "5dba1a20a364f11ae831286e3fd98d36"
server_key_iv: "53578ec9922ec0d377a7fc110a3cce16"
server_key_version: "encryptionkey@0"
I20260619 18:42:48.867275 26670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:48.871585 26670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:48.872179 26670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:48.872354 26670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:48.879330 26670 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:34805
I20260619 18:42:48.879339 26736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:34805 every 8 connection(s)
I20260619 18:42:48.880322 26670 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:42:48.883718 26737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:48.885138 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 26670
I20260619 18:42:48.885270 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:42:48.885593 22077 external_mini_cluster.cc:1428] Setting key 7790300a894edb30c21b024415f3a71c
I20260619 18:42:48.890920 26737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75d768e855db4ac2870dfc64a2ec7ac5: Bootstrap starting.
I20260619 18:42:48.891839 26740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:48.887118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47924 (local address 127.21.143.126:34805)
0619 18:42:48.887805 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:48.887821 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:48.887857 (+ 36us) server_negotiation.cc:407] Connection header received
0619 18:42:48.888697 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:48.888723 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:48.889218 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:48.889622 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:48.890425 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:48.890430 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:48.890457 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:42:48.890468 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:48.890520 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:42:48.890538 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:48.890690 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:48.890764 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 18:42:48.891025 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":155,"threads_started":1}
I20260619 18:42:48.893520 26737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75d768e855db4ac2870dfc64a2ec7ac5: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:48.894381 26737 log.cc:824] T 00000000000000000000000000000000 P 75d768e855db4ac2870dfc64a2ec7ac5: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:48.896323 26737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75d768e855db4ac2870dfc64a2ec7ac5: No bootstrap required, opened a new log
I20260619 18:42:48.899991 26737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75d768e855db4ac2870dfc64a2ec7ac5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75d768e855db4ac2870dfc64a2ec7ac5" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 34805 } }
I20260619 18:42:48.900245 26737 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75d768e855db4ac2870dfc64a2ec7ac5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:48.900328 26737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75d768e855db4ac2870dfc64a2ec7ac5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75d768e855db4ac2870dfc64a2ec7ac5, State: Initialized, Role: FOLLOWER
I20260619 18:42:48.900849 26737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75d768e855db4ac2870dfc64a2ec7ac5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75d768e855db4ac2870dfc64a2ec7ac5" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 34805 } }
I20260619 18:42:48.900983 26737 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75d768e855db4ac2870dfc64a2ec7ac5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:42:48.901032 26737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75d768e855db4ac2870dfc64a2ec7ac5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:42:48.901104 26737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75d768e855db4ac2870dfc64a2ec7ac5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:48.901973 26737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75d768e855db4ac2870dfc64a2ec7ac5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75d768e855db4ac2870dfc64a2ec7ac5" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 34805 } }
I20260619 18:42:48.902271 26737 leader_election.cc:302] T 00000000000000000000000000000000 P 75d768e855db4ac2870dfc64a2ec7ac5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75d768e855db4ac2870dfc64a2ec7ac5; no voters:
I20260619 18:42:48.902531 26737 leader_election.cc:288] T 00000000000000000000000000000000 P 75d768e855db4ac2870dfc64a2ec7ac5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:42:48.902663 26742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75d768e855db4ac2870dfc64a2ec7ac5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:48.902923 26742 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75d768e855db4ac2870dfc64a2ec7ac5 [term 1 LEADER]: Becoming Leader. State: Replica: 75d768e855db4ac2870dfc64a2ec7ac5, State: Running, Role: LEADER
I20260619 18:42:48.903237 26742 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75d768e855db4ac2870dfc64a2ec7ac5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75d768e855db4ac2870dfc64a2ec7ac5" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 34805 } }
I20260619 18:42:48.903825 26737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75d768e855db4ac2870dfc64a2ec7ac5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:42:48.905050 26743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75d768e855db4ac2870dfc64a2ec7ac5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75d768e855db4ac2870dfc64a2ec7ac5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75d768e855db4ac2870dfc64a2ec7ac5" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 34805 } } }
I20260619 18:42:48.905033 26744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75d768e855db4ac2870dfc64a2ec7ac5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75d768e855db4ac2870dfc64a2ec7ac5. Latest consensus state: current_term: 1 leader_uuid: "75d768e855db4ac2870dfc64a2ec7ac5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75d768e855db4ac2870dfc64a2ec7ac5" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 34805 } } }
I20260619 18:42:48.905181 26743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75d768e855db4ac2870dfc64a2ec7ac5 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:48.905203 26744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75d768e855db4ac2870dfc64a2ec7ac5 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:48.905582 26751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:42:48.908820 26751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:42:48.915011 26751 catalog_manager.cc:1347] Generated new cluster ID: c6f877a904c442969236117650118080
I20260619 18:42:48.915094 26751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:42:48.936393 26751 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:42:48.936728 26751 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:42:48.945077 26751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75d768e855db4ac2870dfc64a2ec7ac5: Generated new TSK 0
I20260619 18:42:48.945886 26751 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:42:48.949400 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.65:0
--local_ip_for_outbound_sockets=127.21.143.65
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:34805
--builtin_ntp_servers=127.21.143.84:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 18:42:49.058631 26761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:49.058912 26761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:49.059027 26761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:49.063194 26761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:49.063253 26761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:49.063362 26761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.65
I20260619 18:42:49.068131 26761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:34805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.65
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:49.069213 26761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:49.070643 26761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 18:42:49.080024 26761 server_base.cc:1034] running on GCE node
W20260619 18:42:49.079960 26766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:49.079998 26767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:49.079960 26769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:49.080533 26761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:49.081137 26761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:49.082289 26761 hybrid_clock.cc:648] HybridClock initialized: now 1781894569082261 us; error 48 us; skew 500 ppm
I20260619 18:42:49.084143 26761 webserver.cc:460] Webserver started at http://127.21.143.65:36295/ using document root <none> and password file <none>
I20260619 18:42:49.084714 26761 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:49.084807 26761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:49.085026 26761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:49.086979 26761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-0/data/instance:
uuid: "24a911640df74b5895273e3b60c01b93"
format_stamp: "Formatted at 2026-06-19 18:42:49 on dist-test-slave-mld5"
server_key: "9e4a6a3b9880234f9d42b3769b659e78"
server_key_iv: "68f81bd940170a5a238472213a183540"
server_key_version: "encryptionkey@0"
I20260619 18:42:49.087530 26761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance:
uuid: "24a911640df74b5895273e3b60c01b93"
format_stamp: "Formatted at 2026-06-19 18:42:49 on dist-test-slave-mld5"
server_key: "9e4a6a3b9880234f9d42b3769b659e78"
server_key_iv: "68f81bd940170a5a238472213a183540"
server_key_version: "encryptionkey@0"
I20260619 18:42:49.090893 26761 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:42:49.093364 26775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:49.094563 26761 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:42:49.094703 26761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-0/wal
uuid: "24a911640df74b5895273e3b60c01b93"
format_stamp: "Formatted at 2026-06-19 18:42:49 on dist-test-slave-mld5"
server_key: "9e4a6a3b9880234f9d42b3769b659e78"
server_key_iv: "68f81bd940170a5a238472213a183540"
server_key_version: "encryptionkey@0"
I20260619 18:42:49.094815 26761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:49.099411 26761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:49.099974 26761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:49.100157 26761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:49.100673 26761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:49.101540 26761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:49.101585 26761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:49.101646 26761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:49.101677 26761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:49.110464 26761 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.65:36827
I20260619 18:42:49.110483 26888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.65:36827 every 8 connection(s)
I20260619 18:42:49.111410 26761 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
I20260619 18:42:49.115463 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 26761
I20260619 18:42:49.115583 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance
I20260619 18:42:49.115888 22077 external_mini_cluster.cc:1428] Setting key b4604011b2aa0965b768995cb14fb452
I20260619 18:42:49.117674 26740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:49.113164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:55027 (local address 127.21.143.126:34805)
0619 18:42:49.113300 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:49.113305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:49.114346 (+ 1041us) server_negotiation.cc:407] Connection header received
0619 18:42:49.115768 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:49.115771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:49.115816 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:49.115919 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:49.117191 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:49.117197 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:49.117201 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:42:49.117217 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:49.117240 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:42:49.117250 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:49.117334 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:49.117506 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 18:42:49.117549 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:42:49.118002 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.66:0
--local_ip_for_outbound_sockets=127.21.143.66
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:34805
--builtin_ntp_servers=127.21.143.84:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 18:42:49.118564 26891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:49.113577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:34805 (local address 127.21.143.65:55027)
0619 18:42:49.114198 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:49.114238 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:49.115492 (+ 1254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:49.116118 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:49.116125 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:49.116240 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:49.116849 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:42:49.116852 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:49.116905 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:42:49.116984 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:49.117350 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:49.117359 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:49.117366 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 18:42:49.117543 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 18:42:49.117770 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":102,"threads_started":1}
I20260619 18:42:49.119745 26889 heartbeater.cc:347] Connected to a master server at 127.21.143.126:34805
I20260619 18:42:49.120007 26889 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:49.120468 26889 heartbeater.cc:511] Master 127.21.143.126:34805 requested a full tablet report, sending...
I20260619 18:42:49.122079 26701 ts_manager.cc:194] Registered new tserver with Master: 24a911640df74b5895273e3b60c01b93 (127.21.143.65:36827)
W20260619 18:42:49.228612 26892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:49.228907 26892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:49.228982 26892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:49.232877 26892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:49.232954 26892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:49.233048 26892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.66
I20260619 18:42:49.237627 26892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:34805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.66
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:49.238683 26892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:49.240108 26892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:49.250051 26897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:49.250069 26900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:49.250099 26892 server_base.cc:1034] running on GCE node
W20260619 18:42:49.250054 26898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:49.250811 26892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:49.251590 26892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:49.252753 26892 hybrid_clock.cc:648] HybridClock initialized: now 1781894569252729 us; error 36 us; skew 500 ppm
I20260619 18:42:49.254727 26892 webserver.cc:460] Webserver started at http://127.21.143.66:42293/ using document root <none> and password file <none>
I20260619 18:42:49.255389 26892 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:49.255460 26892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:49.255681 26892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:49.257874 26892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-1/data/instance:
uuid: "705a318efcc44f6994c82ce5ededb7bd"
format_stamp: "Formatted at 2026-06-19 18:42:49 on dist-test-slave-mld5"
server_key: "df54ff1956af2c9c656d35e33c0c00b5"
server_key_iv: "396fededc6de79fec1b24e5da957516e"
server_key_version: "encryptionkey@0"
I20260619 18:42:49.258445 26892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance:
uuid: "705a318efcc44f6994c82ce5ededb7bd"
format_stamp: "Formatted at 2026-06-19 18:42:49 on dist-test-slave-mld5"
server_key: "df54ff1956af2c9c656d35e33c0c00b5"
server_key_iv: "396fededc6de79fec1b24e5da957516e"
server_key_version: "encryptionkey@0"
I20260619 18:42:49.262236 26892 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 18:42:49.264905 26906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:49.266134 26892 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:42:49.266229 26892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-1/wal
uuid: "705a318efcc44f6994c82ce5ededb7bd"
format_stamp: "Formatted at 2026-06-19 18:42:49 on dist-test-slave-mld5"
server_key: "df54ff1956af2c9c656d35e33c0c00b5"
server_key_iv: "396fededc6de79fec1b24e5da957516e"
server_key_version: "encryptionkey@0"
I20260619 18:42:49.266371 26892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:49.272284 26892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:49.272976 26892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:49.273190 26892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:49.273774 26892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:49.274724 26892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:49.274794 26892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:49.274860 26892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:49.274910 26892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:49.283968 26892 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.66:37727
I20260619 18:42:49.283987 27019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.66:37727 every 8 connection(s)
I20260619 18:42:49.284901 26892 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
I20260619 18:42:49.290848 26740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:49.286664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:39805 (local address 127.21.143.126:34805)
0619 18:42:49.286808 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:49.286812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:49.287911 (+ 1099us) server_negotiation.cc:407] Connection header received
0619 18:42:49.289237 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:49.289241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:49.289283 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:49.289364 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:49.290377 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:49.290380 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:49.290383 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:42:49.290396 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:49.290417 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:42:49.290433 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:49.290525 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:49.290672 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 18:42:49.290715 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 18:42:49.291710 27022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:49.287104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:34805 (local address 127.21.143.66:39805)
0619 18:42:49.287769 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:49.287808 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:49.289021 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:49.289538 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:49.289545 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:49.289630 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:49.290126 (+ 496us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:42:49.290127 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:49.290166 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:42:49.290230 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:49.290519 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:49.290527 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:49.290535 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 18:42:49.290703 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 18:42:49.290929 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":153,"threads_started":1}
I20260619 18:42:49.292773 27020 heartbeater.cc:347] Connected to a master server at 127.21.143.126:34805
I20260619 18:42:49.292984 27020 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:49.293359 27020 heartbeater.cc:511] Master 127.21.143.126:34805 requested a full tablet report, sending...
I20260619 18:42:49.294324 26701 ts_manager.cc:194] Registered new tserver with Master: 705a318efcc44f6994c82ce5ededb7bd (127.21.143.66:37727)
I20260619 18:42:49.294647 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 26892
I20260619 18:42:49.294770 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance
I20260619 18:42:49.295115 22077 external_mini_cluster.cc:1428] Setting key f57ed5337c8506b64f471fc916262a9f
I20260619 18:42:49.296823 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.67:0
--local_ip_for_outbound_sockets=127.21.143.67
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:34805
--builtin_ntp_servers=127.21.143.84:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 18:42:49.405161 27023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:49.405537 27023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:49.405622 27023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:49.411388 27023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:49.411500 27023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:49.411602 27023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.67
I20260619 18:42:49.417752 27023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:34805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.67
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:49.418954 27023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:49.420441 27023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:49.429795 27029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:49.429826 27028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:49.429826 27031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:49.430011 27023 server_base.cc:1034] running on GCE node
I20260619 18:42:49.430508 27023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:49.431142 27023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:49.432294 27023 hybrid_clock.cc:648] HybridClock initialized: now 1781894569432256 us; error 58 us; skew 500 ppm
I20260619 18:42:49.434115 27023 webserver.cc:460] Webserver started at http://127.21.143.67:45845/ using document root <none> and password file <none>
I20260619 18:42:49.434702 27023 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:49.434765 27023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:49.434978 27023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:49.437116 27023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-2/data/instance:
uuid: "2b7ff9dcd0964c56a2c1566332b52970"
format_stamp: "Formatted at 2026-06-19 18:42:49 on dist-test-slave-mld5"
server_key: "eee77e678c5794488dc6a374f2b87314"
server_key_iv: "c0d6edb76a63d3782b7d3beb3c48ff23"
server_key_version: "encryptionkey@0"
I20260619 18:42:49.437692 27023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance:
uuid: "2b7ff9dcd0964c56a2c1566332b52970"
format_stamp: "Formatted at 2026-06-19 18:42:49 on dist-test-slave-mld5"
server_key: "eee77e678c5794488dc6a374f2b87314"
server_key_iv: "c0d6edb76a63d3782b7d3beb3c48ff23"
server_key_version: "encryptionkey@0"
I20260619 18:42:49.441459 27023 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:42:49.444020 27037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:49.445418 27023 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:42:49.445511 27023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-2/wal
uuid: "2b7ff9dcd0964c56a2c1566332b52970"
format_stamp: "Formatted at 2026-06-19 18:42:49 on dist-test-slave-mld5"
server_key: "eee77e678c5794488dc6a374f2b87314"
server_key_iv: "c0d6edb76a63d3782b7d3beb3c48ff23"
server_key_version: "encryptionkey@0"
I20260619 18:42:49.445601 27023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:49.450428 27023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:49.450976 27023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:49.451104 27023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:49.451646 27023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:49.452551 27023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:49.452596 27023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:49.452633 27023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:49.452654 27023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:49.461187 27023 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.67:36291
I20260619 18:42:49.461225 27150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.67:36291 every 8 connection(s)
I20260619 18:42:49.462172 27023 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
I20260619 18:42:49.462399 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 27023
I20260619 18:42:49.462522 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance
I20260619 18:42:49.462867 22077 external_mini_cluster.cc:1428] Setting key c4cd544da67dbe62a7ec895ed892593e
I20260619 18:42:49.468274 26740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:49.464199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:44667 (local address 127.21.143.126:34805)
0619 18:42:49.464343 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:49.464348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:49.465543 (+ 1195us) server_negotiation.cc:407] Connection header received
0619 18:42:49.466816 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:49.466818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:49.466874 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:49.466942 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:49.467932 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:49.467935 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:49.467937 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:42:49.467946 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:49.467963 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:42:49.467970 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:49.468043 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:49.468140 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 18:42:49.468174 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 18:42:49.469095 27153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:49.464667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:34805 (local address 127.21.143.67:44667)
0619 18:42:49.465417 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:49.465458 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:49.466622 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:49.467070 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:49.467076 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:49.467162 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:49.467696 (+ 534us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:42:49.467698 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:49.467737 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:42:49.467802 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:49.468035 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:49.468039 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:49.468042 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 18:42:49.468157 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 18:42:49.468336 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":118,"threads_started":1}
I20260619 18:42:49.470080 27151 heartbeater.cc:347] Connected to a master server at 127.21.143.126:34805
I20260619 18:42:49.470322 27151 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:49.470732 27151 heartbeater.cc:511] Master 127.21.143.126:34805 requested a full tablet report, sending...
I20260619 18:42:49.471742 26701 ts_manager.cc:194] Registered new tserver with Master: 2b7ff9dcd0964c56a2c1566332b52970 (127.21.143.67:36291)
I20260619 18:42:49.476763 22077 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:42:49.479928 26740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:49.478484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47932 (local address 127.21.143.126:34805)
0619 18:42:49.478589 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:49.478592 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:49.478737 (+ 145us) server_negotiation.cc:407] Connection header received
0619 18:42:49.478821 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:49.478824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:49.478850 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:49.478912 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:49.479632 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:49.479635 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:49.479637 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:42:49.479646 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:49.479661 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:42:49.479668 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:49.479733 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:49.479807 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 18:42:49.479838 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 18:42:49.482355 26701 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:42:49.485332 26701 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:42:49.495434 27162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:49.492547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:36827 (local address 127.0.0.1:34940)
0619 18:42:49.492939 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:49.492988 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:49.493082 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:49.494038 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:49.494043 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:49.494110 (+ 67us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:49.494616 (+ 506us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:42:49.494620 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:49.494642 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:42:49.494711 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:49.495063 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:49.495071 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:49.495076 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 18:42:49.495181 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 18:42:49.495248 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":54,"threads_started":1}
I20260619 18:42:49.495525 27166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:49.493450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53584 (local address 127.21.143.66:37727)
0619 18:42:49.493834 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:49.493844 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:49.493866 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:42:49.493922 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:49.493930 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:49.494094 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:49.494225 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:49.494986 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:49.494992 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:49.495003 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:42:49.495014 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:49.495059 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:42:49.495090 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:49.495195 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:49.495291 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 18:42:49.495387 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":110,"threads_started":1}
I20260619 18:42:49.495589 27163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:49.492670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34940 (local address 127.21.143.65:36827)
0619 18:42:49.493583 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:49.493595 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:49.493620 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:42:49.493669 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:49.493675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:49.493859 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:49.494354 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:49.494865 (+ 511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:49.494872 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:49.494895 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:42:49.494907 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:49.494946 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:42:49.494988 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:49.495095 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:49.495398 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 18:42:49.495453 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":772,"thread_start_us":122,"threads_started":1}
I20260619 18:42:49.496213 27164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:49.492841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:37727 (local address 127.0.0.1:53584)
0619 18:42:49.493256 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:49.493268 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:49.493329 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:49.494218 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:49.494221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:49.494233 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:49.494838 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:42:49.494840 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:49.494849 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:42:49.494867 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:49.495188 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:49.495191 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:49.495194 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 18:42:49.495269 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 18:42:49.495328 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":90,"threads_started":1}
I20260619 18:42:49.497465 27165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:49.493264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:36291 (local address 127.0.0.1:38746)
0619 18:42:49.494939 (+ 1675us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:49.494952 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:49.495027 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:49.495845 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:49.495847 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:49.495859 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:49.496605 (+ 746us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:42:49.496607 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:49.496616 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:42:49.496640 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:49.497234 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:49.497237 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:49.497240 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 18:42:49.497322 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 18:42:49.497356 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1614,"thread_start_us":102,"threads_started":1}
I20260619 18:42:49.497553 27167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:49.493422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38746 (local address 127.21.143.67:36291)
0619 18:42:49.495257 (+ 1835us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:49.495268 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:49.495290 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:42:49.495357 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:49.495364 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:49.495534 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:49.495848 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:49.497070 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:49.497075 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:49.497084 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:42:49.497095 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:49.497133 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:42:49.497159 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:49.497256 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:49.497345 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 18:42:49.497414 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1737,"thread_start_us":167,"threads_started":1}
I20260619 18:42:49.498807 26954 tablet_service.cc:1467] Processing CreateTablet for tablet 4abd3f8f287644448d0aebbc71ef7ac9 (DEFAULT_TABLE table=test-table [id=260edb5cfad546f0b72a841035650c18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:42:49.498806 26823 tablet_service.cc:1467] Processing CreateTablet for tablet 4abd3f8f287644448d0aebbc71ef7ac9 (DEFAULT_TABLE table=test-table [id=260edb5cfad546f0b72a841035650c18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:42:49.499926 26954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4abd3f8f287644448d0aebbc71ef7ac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:49.499926 26823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4abd3f8f287644448d0aebbc71ef7ac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:49.500118 27085 tablet_service.cc:1467] Processing CreateTablet for tablet 4abd3f8f287644448d0aebbc71ef7ac9 (DEFAULT_TABLE table=test-table [id=260edb5cfad546f0b72a841035650c18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:42:49.501025 27085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4abd3f8f287644448d0aebbc71ef7ac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:49.506773 27168 tablet_bootstrap.cc:492] T 4abd3f8f287644448d0aebbc71ef7ac9 P 2b7ff9dcd0964c56a2c1566332b52970: Bootstrap starting.
I20260619 18:42:49.507447 27170 tablet_bootstrap.cc:492] T 4abd3f8f287644448d0aebbc71ef7ac9 P 705a318efcc44f6994c82ce5ededb7bd: Bootstrap starting.
I20260619 18:42:49.507923 27169 tablet_bootstrap.cc:492] T 4abd3f8f287644448d0aebbc71ef7ac9 P 24a911640df74b5895273e3b60c01b93: Bootstrap starting.
I20260619 18:42:49.509665 27170 tablet_bootstrap.cc:654] T 4abd3f8f287644448d0aebbc71ef7ac9 P 705a318efcc44f6994c82ce5ededb7bd: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:49.509701 27168 tablet_bootstrap.cc:654] T 4abd3f8f287644448d0aebbc71ef7ac9 P 2b7ff9dcd0964c56a2c1566332b52970: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:49.509768 27169 tablet_bootstrap.cc:654] T 4abd3f8f287644448d0aebbc71ef7ac9 P 24a911640df74b5895273e3b60c01b93: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:49.510445 27169 log.cc:824] T 4abd3f8f287644448d0aebbc71ef7ac9 P 24a911640df74b5895273e3b60c01b93: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:49.510516 27168 log.cc:824] T 4abd3f8f287644448d0aebbc71ef7ac9 P 2b7ff9dcd0964c56a2c1566332b52970: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:49.510516 27170 log.cc:824] T 4abd3f8f287644448d0aebbc71ef7ac9 P 705a318efcc44f6994c82ce5ededb7bd: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:49.512274 27169 tablet_bootstrap.cc:492] T 4abd3f8f287644448d0aebbc71ef7ac9 P 24a911640df74b5895273e3b60c01b93: No bootstrap required, opened a new log
I20260619 18:42:49.512274 27168 tablet_bootstrap.cc:492] T 4abd3f8f287644448d0aebbc71ef7ac9 P 2b7ff9dcd0964c56a2c1566332b52970: No bootstrap required, opened a new log
I20260619 18:42:49.512463 27169 ts_tablet_manager.cc:1397] T 4abd3f8f287644448d0aebbc71ef7ac9 P 24a911640df74b5895273e3b60c01b93: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 18:42:49.512472 27168 ts_tablet_manager.cc:1397] T 4abd3f8f287644448d0aebbc71ef7ac9 P 2b7ff9dcd0964c56a2c1566332b52970: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260619 18:42:49.513501 27170 tablet_bootstrap.cc:492] T 4abd3f8f287644448d0aebbc71ef7ac9 P 705a318efcc44f6994c82ce5ededb7bd: No bootstrap required, opened a new log
I20260619 18:42:49.513664 27170 ts_tablet_manager.cc:1397] T 4abd3f8f287644448d0aebbc71ef7ac9 P 705a318efcc44f6994c82ce5ededb7bd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 18:42:49.516093 27169 raft_consensus.cc:348] T 4abd3f8f287644448d0aebbc71ef7ac9 P 24a911640df74b5895273e3b60c01b93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a911640df74b5895273e3b60c01b93" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36827 } } peers { permanent_uuid: "2b7ff9dcd0964c56a2c1566332b52970" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 36291 } } peers { permanent_uuid: "705a318efcc44f6994c82ce5ededb7bd" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 37727 } }
I20260619 18:42:49.516327 27169 raft_consensus.cc:374] T 4abd3f8f287644448d0aebbc71ef7ac9 P 24a911640df74b5895273e3b60c01b93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:49.516381 27169 raft_consensus.cc:729] T 4abd3f8f287644448d0aebbc71ef7ac9 P 24a911640df74b5895273e3b60c01b93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a911640df74b5895273e3b60c01b93, State: Initialized, Role: FOLLOWER
I20260619 18:42:49.516808 27169 consensus_queue.cc:260] T 4abd3f8f287644448d0aebbc71ef7ac9 P 24a911640df74b5895273e3b60c01b93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a911640df74b5895273e3b60c01b93" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36827 } } peers { permanent_uuid: "2b7ff9dcd0964c56a2c1566332b52970" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 36291 } } peers { permanent_uuid: "705a318efcc44f6994c82ce5ededb7bd" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 37727 } }
I20260619 18:42:49.516724 27168 raft_consensus.cc:348] T 4abd3f8f287644448d0aebbc71ef7ac9 P 2b7ff9dcd0964c56a2c1566332b52970 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a911640df74b5895273e3b60c01b93" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36827 } } peers { permanent_uuid: "2b7ff9dcd0964c56a2c1566332b52970" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 36291 } } peers { permanent_uuid: "705a318efcc44f6994c82ce5ededb7bd" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 37727 } }
I20260619 18:42:49.516965 27168 raft_consensus.cc:374] T 4abd3f8f287644448d0aebbc71ef7ac9 P 2b7ff9dcd0964c56a2c1566332b52970 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:49.517027 27168 raft_consensus.cc:729] T 4abd3f8f287644448d0aebbc71ef7ac9 P 2b7ff9dcd0964c56a2c1566332b52970 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b7ff9dcd0964c56a2c1566332b52970, State: Initialized, Role: FOLLOWER
I20260619 18:42:49.517433 27168 consensus_queue.cc:260] T 4abd3f8f287644448d0aebbc71ef7ac9 P 2b7ff9dcd0964c56a2c1566332b52970 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a911640df74b5895273e3b60c01b93" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36827 } } peers { permanent_uuid: "2b7ff9dcd0964c56a2c1566332b52970" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 36291 } } peers { permanent_uuid: "705a318efcc44f6994c82ce5ededb7bd" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 37727 } }
I20260619 18:42:49.517768 27169 ts_tablet_manager.cc:1428] T 4abd3f8f287644448d0aebbc71ef7ac9 P 24a911640df74b5895273e3b60c01b93: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260619 18:42:49.518101 27151 heartbeater.cc:503] Master 127.21.143.126:34805 was elected leader, sending a full tablet report...
I20260619 18:42:49.518106 26889 heartbeater.cc:503] Master 127.21.143.126:34805 was elected leader, sending a full tablet report...
I20260619 18:42:49.518329 27168 ts_tablet_manager.cc:1428] T 4abd3f8f287644448d0aebbc71ef7ac9 P 2b7ff9dcd0964c56a2c1566332b52970: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260619 18:42:49.517959 27170 raft_consensus.cc:348] T 4abd3f8f287644448d0aebbc71ef7ac9 P 705a318efcc44f6994c82ce5ededb7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a911640df74b5895273e3b60c01b93" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36827 } } peers { permanent_uuid: "2b7ff9dcd0964c56a2c1566332b52970" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 36291 } } peers { permanent_uuid: "705a318efcc44f6994c82ce5ededb7bd" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 37727 } }
I20260619 18:42:49.518738 27170 raft_consensus.cc:374] T 4abd3f8f287644448d0aebbc71ef7ac9 P 705a318efcc44f6994c82ce5ededb7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:49.518796 27170 raft_consensus.cc:729] T 4abd3f8f287644448d0aebbc71ef7ac9 P 705a318efcc44f6994c82ce5ededb7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 705a318efcc44f6994c82ce5ededb7bd, State: Initialized, Role: FOLLOWER
I20260619 18:42:49.519187 27170 consensus_queue.cc:260] T 4abd3f8f287644448d0aebbc71ef7ac9 P 705a318efcc44f6994c82ce5ededb7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a911640df74b5895273e3b60c01b93" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36827 } } peers { permanent_uuid: "2b7ff9dcd0964c56a2c1566332b52970" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 36291 } } peers { permanent_uuid: "705a318efcc44f6994c82ce5ededb7bd" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 37727 } }
I20260619 18:42:49.519794 27020 heartbeater.cc:503] Master 127.21.143.126:34805 was elected leader, sending a full tablet report...
I20260619 18:42:49.520093 27170 ts_tablet_manager.cc:1428] T 4abd3f8f287644448d0aebbc71ef7ac9 P 705a318efcc44f6994c82ce5ededb7bd: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
W20260619 18:42:49.535794 27021 tablet.cc:2369] T 4abd3f8f287644448d0aebbc71ef7ac9 P 705a318efcc44f6994c82ce5ededb7bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:42:49.612546 26890 tablet.cc:2369] T 4abd3f8f287644448d0aebbc71ef7ac9 P 24a911640df74b5895273e3b60c01b93: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:42:49.675235 27175 raft_consensus.cc:482] T 4abd3f8f287644448d0aebbc71ef7ac9 P 2b7ff9dcd0964c56a2c1566332b52970 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:42:49.675513 27175 raft_consensus.cc:504] T 4abd3f8f287644448d0aebbc71ef7ac9 P 2b7ff9dcd0964c56a2c1566332b52970 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a911640df74b5895273e3b60c01b93" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36827 } } peers { permanent_uuid: "2b7ff9dcd0964c56a2c1566332b52970" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 36291 } } peers { permanent_uuid: "705a318efcc44f6994c82ce5ededb7bd" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 37727 } }
I20260619 18:42:49.676698 27175 leader_election.cc:288] T 4abd3f8f287644448d0aebbc71ef7ac9 P 2b7ff9dcd0964c56a2c1566332b52970 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24a911640df74b5895273e3b60c01b93 (127.21.143.65:36827), 705a318efcc44f6994c82ce5ededb7bd (127.21.143.66:37727)
I20260619 18:42:49.679057 27166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:49.677275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:56367 (local address 127.21.143.66:37727)
0619 18:42:49.677417 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:49.677423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:49.677516 (+ 93us) server_negotiation.cc:407] Connection header received
0619 18:42:49.677630 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:49.677633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:49.677674 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:49.677734 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:49.678612 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:49.678617 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:49.678620 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:42:49.678632 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:49.678652 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:42:49.678661 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:49.678756 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:49.678886 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 18:42:49.678925 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 18:42:49.679057 27153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:49.676999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:36827 (local address 127.21.143.67:52829)
0619 18:42:49.677364 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:49.677382 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:49.677509 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:49.677815 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:49.677817 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:49.677830 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:49.678522 (+ 692us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:42:49.678524 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:49.678534 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:42:49.678558 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:49.678807 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:49.678810 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:49.678813 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 18:42:49.678885 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 18:42:49.678919 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255}
I20260619 18:42:49.679085 27177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:49.677032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:37727 (local address 127.21.143.67:56367)
0619 18:42:49.677393 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:49.677411 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:49.677539 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:49.677779 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:49.677783 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:49.677797 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:49.678394 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:42:49.678398 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:49.678412 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:42:49.678443 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:49.678770 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:49.678773 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:49.678776 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 18:42:49.678856 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 18:42:49.678893 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":107,"threads_started":1}
I20260619 18:42:49.679477 27163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:49.677091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:52829 (local address 127.21.143.65:36827)
0619 18:42:49.677224 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:49.677229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:49.677471 (+ 242us) server_negotiation.cc:407] Connection header received
0619 18:42:49.677649 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:49.677652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:49.677684 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:49.677777 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:49.678654 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:49.678658 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:49.678660 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:42:49.678670 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:49.678690 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:42:49.678701 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:49.678780 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:49.679264 (+ 484us) server_negotiation.cc:299] Negotiation successful
0619 18:42:49.679353 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 18:42:49.680090 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4abd3f8f287644448d0aebbc71ef7ac9" candidate_uuid: "2b7ff9dcd0964c56a2c1566332b52970" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "705a318efcc44f6994c82ce5ededb7bd" is_pre_election: true
I20260619 18:42:49.680325 26843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4abd3f8f287644448d0aebbc71ef7ac9" candidate_uuid: "2b7ff9dcd0964c56a2c1566332b52970" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a911640df74b5895273e3b60c01b93" is_pre_election: true
I20260619 18:42:49.680433 26974 raft_consensus.cc:2454] T 4abd3f8f287644448d0aebbc71ef7ac9 P 705a318efcc44f6994c82ce5ededb7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b7ff9dcd0964c56a2c1566332b52970 in term 0.
I20260619 18:42:49.680650 26843 raft_consensus.cc:2454] T 4abd3f8f287644448d0aebbc71ef7ac9 P 24a911640df74b5895273e3b60c01b93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b7ff9dcd0964c56a2c1566332b52970 in term 0.
I20260619 18:42:49.680972 27039 leader_election.cc:302] T 4abd3f8f287644448d0aebbc71ef7ac9 P 2b7ff9dcd0964c56a2c1566332b52970 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b7ff9dcd0964c56a2c1566332b52970, 705a318efcc44f6994c82ce5ededb7bd; no voters:
I20260619 18:42:49.681250 27175 raft_consensus.cc:2781] T 4abd3f8f287644448d0aebbc71ef7ac9 P 2b7ff9dcd0964c56a2c1566332b52970 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:42:49.681332 27175 raft_consensus.cc:482] T 4abd3f8f287644448d0aebbc71ef7ac9 P 2b7ff9dcd0964c56a2c1566332b52970 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:42:49.681381 27175 raft_consensus.cc:3037] T 4abd3f8f287644448d0aebbc71ef7ac9 P 2b7ff9dcd0964c56a2c1566332b52970 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:49.682430 27175 raft_consensus.cc:504] T 4abd3f8f287644448d0aebbc71ef7ac9 P 2b7ff9dcd0964c56a2c1566332b52970 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a911640df74b5895273e3b60c01b93" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36827 } } peers { permanent_uuid: "2b7ff9dcd0964c56a2c1566332b52970" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 36291 } } peers { permanent_uuid: "705a318efcc44f6994c82ce5ededb7bd" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 37727 } }
I20260619 18:42:49.682776 27175 leader_election.cc:288] T 4abd3f8f287644448d0aebbc71ef7ac9 P 2b7ff9dcd0964c56a2c1566332b52970 [CANDIDATE]: Term 1 election: Requested vote from peers 24a911640df74b5895273e3b60c01b93 (127.21.143.65:36827), 705a318efcc44f6994c82ce5ededb7bd (127.21.143.66:37727)
I20260619 18:42:49.683050 26843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4abd3f8f287644448d0aebbc71ef7ac9" candidate_uuid: "2b7ff9dcd0964c56a2c1566332b52970" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a911640df74b5895273e3b60c01b93"
I20260619 18:42:49.683089 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4abd3f8f287644448d0aebbc71ef7ac9" candidate_uuid: "2b7ff9dcd0964c56a2c1566332b52970" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "705a318efcc44f6994c82ce5ededb7bd"
I20260619 18:42:49.683156 26843 raft_consensus.cc:3037] T 4abd3f8f287644448d0aebbc71ef7ac9 P 24a911640df74b5895273e3b60c01b93 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:49.683225 26974 raft_consensus.cc:3037] T 4abd3f8f287644448d0aebbc71ef7ac9 P 705a318efcc44f6994c82ce5ededb7bd [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:49.684183 26843 raft_consensus.cc:2454] T 4abd3f8f287644448d0aebbc71ef7ac9 P 24a911640df74b5895273e3b60c01b93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b7ff9dcd0964c56a2c1566332b52970 in term 1.
I20260619 18:42:49.684346 26974 raft_consensus.cc:2454] T 4abd3f8f287644448d0aebbc71ef7ac9 P 705a318efcc44f6994c82ce5ededb7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b7ff9dcd0964c56a2c1566332b52970 in term 1.
I20260619 18:42:49.684645 27039 leader_election.cc:302] T 4abd3f8f287644448d0aebbc71ef7ac9 P 2b7ff9dcd0964c56a2c1566332b52970 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b7ff9dcd0964c56a2c1566332b52970, 705a318efcc44f6994c82ce5ededb7bd; no voters:
I20260619 18:42:49.684809 27175 raft_consensus.cc:2781] T 4abd3f8f287644448d0aebbc71ef7ac9 P 2b7ff9dcd0964c56a2c1566332b52970 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:49.685012 27175 raft_consensus.cc:686] T 4abd3f8f287644448d0aebbc71ef7ac9 P 2b7ff9dcd0964c56a2c1566332b52970 [term 1 LEADER]: Becoming Leader. State: Replica: 2b7ff9dcd0964c56a2c1566332b52970, State: Running, Role: LEADER
I20260619 18:42:49.685341 27175 consensus_queue.cc:237] T 4abd3f8f287644448d0aebbc71ef7ac9 P 2b7ff9dcd0964c56a2c1566332b52970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a911640df74b5895273e3b60c01b93" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36827 } } peers { permanent_uuid: "2b7ff9dcd0964c56a2c1566332b52970" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 36291 } } peers { permanent_uuid: "705a318efcc44f6994c82ce5ededb7bd" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 37727 } }
I20260619 18:42:49.688361 26701 catalog_manager.cc:5495] T 4abd3f8f287644448d0aebbc71ef7ac9 P 2b7ff9dcd0964c56a2c1566332b52970 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b7ff9dcd0964c56a2c1566332b52970 (127.21.143.67). New cstate: current_term: 1 leader_uuid: "2b7ff9dcd0964c56a2c1566332b52970" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24a911640df74b5895273e3b60c01b93" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b7ff9dcd0964c56a2c1566332b52970" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 36291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "705a318efcc44f6994c82ce5ededb7bd" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 37727 } health_report { overall_health: UNKNOWN } } }
W20260619 18:42:49.713217 27152 tablet.cc:2369] T 4abd3f8f287644448d0aebbc71ef7ac9 P 2b7ff9dcd0964c56a2c1566332b52970: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:42:49.731091 27167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:49.729750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38748 (local address 127.21.143.67:36291)
0619 18:42:49.729875 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:49.729879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:49.729893 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:42:49.729991 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:49.729993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:49.730051 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:49.730137 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:49.730776 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:49.730779 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:49.730781 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:42:49.730792 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:49.730810 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:42:49.730818 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:49.730893 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:49.730955 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 18:42:49.730989 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 18:42:49.736778 26974 raft_consensus.cc:1261] T 4abd3f8f287644448d0aebbc71ef7ac9 P 705a318efcc44f6994c82ce5ededb7bd [term 1 FOLLOWER]: Refusing update from remote peer 2b7ff9dcd0964c56a2c1566332b52970: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:42:49.736788 26843 raft_consensus.cc:1261] T 4abd3f8f287644448d0aebbc71ef7ac9 P 24a911640df74b5895273e3b60c01b93 [term 1 FOLLOWER]: Refusing update from remote peer 2b7ff9dcd0964c56a2c1566332b52970: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:42:49.737447 27175 consensus_queue.cc:1035] T 4abd3f8f287644448d0aebbc71ef7ac9 P 2b7ff9dcd0964c56a2c1566332b52970 [LEADER]: Connected to new peer: Peer: permanent_uuid: "705a318efcc44f6994c82ce5ededb7bd" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:42:49.737694 27178 consensus_queue.cc:1035] T 4abd3f8f287644448d0aebbc71ef7ac9 P 2b7ff9dcd0964c56a2c1566332b52970 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24a911640df74b5895273e3b60c01b93" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:42:49.745887 27183 mvcc.cc:205] Tried to move back new op lower bound from 7298640157634289664 to 7298640157433749504. Current Snapshot: MvccSnapshot[applied={T|T < 7298640157634289664}]
I20260619 18:42:49.747427 27182 mvcc.cc:205] Tried to move back new op lower bound from 7298640157634289664 to 7298640157433749504. Current Snapshot: MvccSnapshot[applied={T|T < 7298640157634289664}]
I20260619 18:42:49.753228 26699 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:42:49.753429 26699 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:42:49.756058 26699 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75d768e855db4ac2870dfc64a2ec7ac5: Sending DeleteTablet for 3 replicas of tablet 4abd3f8f287644448d0aebbc71ef7ac9
I20260619 18:42:49.756946 27085 tablet_service.cc:1514] Processing DeleteTablet for tablet 4abd3f8f287644448d0aebbc71ef7ac9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:42:49 UTC) from {username='slave'} at 127.0.0.1:38746
I20260619 18:42:49.756968 26954 tablet_service.cc:1514] Processing DeleteTablet for tablet 4abd3f8f287644448d0aebbc71ef7ac9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:42:49 UTC) from {username='slave'} at 127.0.0.1:53584
I20260619 18:42:49.756960 26823 tablet_service.cc:1514] Processing DeleteTablet for tablet 4abd3f8f287644448d0aebbc71ef7ac9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:42:49 UTC) from {username='slave'} at 127.0.0.1:34940
I20260619 18:42:49.757520 27194 tablet_replica.cc:330] T 4abd3f8f287644448d0aebbc71ef7ac9 P 2b7ff9dcd0964c56a2c1566332b52970: stopping tablet replica
I20260619 18:42:49.757881 27194 raft_consensus.cc:2229] T 4abd3f8f287644448d0aebbc71ef7ac9 P 2b7ff9dcd0964c56a2c1566332b52970 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:42:49.758083 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 26761
I20260619 18:42:49.758879 27193 tablet_replica.cc:330] T 4abd3f8f287644448d0aebbc71ef7ac9 P 705a318efcc44f6994c82ce5ededb7bd: stopping tablet replica
I20260619 18:42:49.759181 27193 raft_consensus.cc:2229] T 4abd3f8f287644448d0aebbc71ef7ac9 P 705a318efcc44f6994c82ce5ededb7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:42:49.759521 27193 raft_consensus.cc:2258] T 4abd3f8f287644448d0aebbc71ef7ac9 P 705a318efcc44f6994c82ce5ededb7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:42:49.758318 27194 raft_consensus.cc:2258] T 4abd3f8f287644448d0aebbc71ef7ac9 P 2b7ff9dcd0964c56a2c1566332b52970 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:42:49.761343 27194 ts_tablet_manager.cc:1905] T 4abd3f8f287644448d0aebbc71ef7ac9 P 2b7ff9dcd0964c56a2c1566332b52970: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:42:49.762282 27193 ts_tablet_manager.cc:1905] T 4abd3f8f287644448d0aebbc71ef7ac9 P 705a318efcc44f6994c82ce5ededb7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:42:49.764400 27194 ts_tablet_manager.cc:1918] T 4abd3f8f287644448d0aebbc71ef7ac9 P 2b7ff9dcd0964c56a2c1566332b52970: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:42:49.764500 27194 log.cc:1196] T 4abd3f8f287644448d0aebbc71ef7ac9 P 2b7ff9dcd0964c56a2c1566332b52970: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-2/wal/wals/4abd3f8f287644448d0aebbc71ef7ac9
I20260619 18:42:49.764833 27194 ts_tablet_manager.cc:1939] T 4abd3f8f287644448d0aebbc71ef7ac9 P 2b7ff9dcd0964c56a2c1566332b52970: Deleting consensus metadata
I20260619 18:42:49.766067 26685 catalog_manager.cc:4841] TS 2b7ff9dcd0964c56a2c1566332b52970 (127.21.143.67:36291): tablet 4abd3f8f287644448d0aebbc71ef7ac9 (table test-table [id=260edb5cfad546f0b72a841035650c18]) successfully deleted
I20260619 18:42:49.766103 27193 ts_tablet_manager.cc:1918] T 4abd3f8f287644448d0aebbc71ef7ac9 P 705a318efcc44f6994c82ce5ededb7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:42:49.766297 27193 log.cc:1196] T 4abd3f8f287644448d0aebbc71ef7ac9 P 705a318efcc44f6994c82ce5ededb7bd: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894542572984-22077-0/minicluster-data/ts-1/wal/wals/4abd3f8f287644448d0aebbc71ef7ac9
I20260619 18:42:49.766676 27193 ts_tablet_manager.cc:1939] T 4abd3f8f287644448d0aebbc71ef7ac9 P 705a318efcc44f6994c82ce5ededb7bd: Deleting consensus metadata
W20260619 18:42:49.766991 26685 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.143.65:36827 (error 108)
I20260619 18:42:49.767195 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 26892
I20260619 18:42:49.772123 27165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:49.771842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:36827 (local address 127.0.0.1:34956)
0619 18:42:49.771975 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:49.772049 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.65:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 18:42:49.772313 26685 catalog_manager.cc:4584] TS 24a911640df74b5895273e3b60c01b93 (127.21.143.65:36827): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4abd3f8f287644448d0aebbc71ef7ac9: Network error: Client connection negotiation failed: client connection to 127.21.143.65:36827: connect: Connection refused (error 111)
I20260619 18:42:49.775128 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 27023
I20260619 18:42:49.775776 27165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:49.775465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:37727 (local address 127.0.0.1:53590)
0619 18:42:49.775631 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:49.775691 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.66:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 18:42:49.780685 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 26670
2026-06-19T18:42:49Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1089 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 18:42:49.801416 22077 mini_oidc.cc:152] Starting JWKS server
I20260619 18:42:49.803054 22077 webserver.cc:460] Webserver started at https://127.0.0.1:33661/ using document root <none> and password file <none>
I20260619 18:42:49.803239 22077 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 18:42:49.803680 22077 webserver.cc:460] Webserver started at http://127.0.0.1:43203/ using document root <none> and password file <none>
2026-06-19T18:42:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:42:49Z Disabled control of system clock
I20260619 18:42:49.812556 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:43007
--webserver_interface=127.21.143.126
--webserver_port=0
--builtin_ntp_servers=127.21.143.84:45641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:43007
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33661/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894542572984-22077-0/testchainca.cert with env {}
W20260619 18:42:49.921237 27201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:49.921517 27201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:49.921558 27201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:49.925612 27201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:42:49.925674 27201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:49.925689 27201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:49.925706 27201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:42:49.925721 27201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 18:42:49.925737 27201 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 18:42:49.925750 27201 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33661/jwks.json
I20260619 18:42:49.931093 27201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:45641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:43007
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:43007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33661/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894542572984-22077-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:49.932179 27201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:49.933163 27201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:49.941321 27207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:49.941290 27209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:49.941470 27206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:49.941576 27201 server_base.cc:1034] running on GCE node
I20260619 18:42:49.942142 27201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:49.943248 27201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:49.944468 27201 hybrid_clock.cc:648] HybridClock initialized: now 1781894569944440 us; error 36 us; skew 500 ppm
I20260619 18:42:49.946204 27201 webserver.cc:460] Webserver started at http://127.21.143.126:41563/ using document root <none> and password file <none>
I20260619 18:42:49.946823 27201 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:49.946898 27201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:49.947119 27201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:49.949313 27201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "fef1f5fd89df475785d1581e53b22d43"
format_stamp: "Formatted at 2026-06-19 18:42:49 on dist-test-slave-mld5"
server_key: "a44f2cb546396e99440f037fc8e3990e"
server_key_iv: "c070620fd46a1c470ad3450ed7ac61a8"
server_key_version: "encryptionkey@0"
I20260619 18:42:49.950079 27201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "fef1f5fd89df475785d1581e53b22d43"
format_stamp: "Formatted at 2026-06-19 18:42:49 on dist-test-slave-mld5"
server_key: "a44f2cb546396e99440f037fc8e3990e"
server_key_iv: "c070620fd46a1c470ad3450ed7ac61a8"
server_key_version: "encryptionkey@0"
I20260619 18:42:49.953970 27201 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260619 18:42:49.956615 27215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:49.957971 27201 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:42:49.958118 27201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "fef1f5fd89df475785d1581e53b22d43"
format_stamp: "Formatted at 2026-06-19 18:42:49 on dist-test-slave-mld5"
server_key: "a44f2cb546396e99440f037fc8e3990e"
server_key_iv: "c070620fd46a1c470ad3450ed7ac61a8"
server_key_version: "encryptionkey@0"
I20260619 18:42:49.958243 27201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:49.984771 27201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:49.985545 27201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:49.985744 27201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:49.992728 27201 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:43007
I20260619 18:42:49.992740 27267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:43007 every 8 connection(s)
I20260619 18:42:49.993657 27201 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:42:49.996906 27268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:49.998294 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 27201
I20260619 18:42:49.998422 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:42:49.998765 22077 external_mini_cluster.cc:1428] Setting key 8e65069f6c1344b36e252955e2c9b324
I20260619 18:42:50.003471 27268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fef1f5fd89df475785d1581e53b22d43: Bootstrap starting.
I20260619 18:42:50.005882 27268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fef1f5fd89df475785d1581e53b22d43: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:50.006639 27268 log.cc:824] T 00000000000000000000000000000000 P fef1f5fd89df475785d1581e53b22d43: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:50.008004 27271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:50.000160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47602 (local address 127.21.143.126:43007)
0619 18:42:50.000789 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:50.000804 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:50.000841 (+ 37us) server_negotiation.cc:407] Connection header received
0619 18:42:50.001486 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:50.001504 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:50.001912 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:50.002205 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:50.002576 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:50.003657 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:50.004569 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:50.005078 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:50.005255 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:50.006138 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:50.006146 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:50.006165 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:42:50.006563 (+ 398us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:50.006617 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:42:50.006637 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:50.006746 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:50.006872 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 18:42:50.007154 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":133,"threads_started":1}
I20260619 18:42:50.008777 27268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fef1f5fd89df475785d1581e53b22d43: No bootstrap required, opened a new log
I20260619 18:42:50.010659 22077 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:42:50.012735 27268 raft_consensus.cc:348] T 00000000000000000000000000000000 P fef1f5fd89df475785d1581e53b22d43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef1f5fd89df475785d1581e53b22d43" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 43007 } }
I20260619 18:42:50.012974 27268 raft_consensus.cc:374] T 00000000000000000000000000000000 P fef1f5fd89df475785d1581e53b22d43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:50.013051 27268 raft_consensus.cc:729] T 00000000000000000000000000000000 P fef1f5fd89df475785d1581e53b22d43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef1f5fd89df475785d1581e53b22d43, State: Initialized, Role: FOLLOWER
I20260619 18:42:50.013571 27268 consensus_queue.cc:260] T 00000000000000000000000000000000 P fef1f5fd89df475785d1581e53b22d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef1f5fd89df475785d1581e53b22d43" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 43007 } }
I20260619 18:42:50.013718 27268 raft_consensus.cc:388] T 00000000000000000000000000000000 P fef1f5fd89df475785d1581e53b22d43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:42:50.013785 27268 raft_consensus.cc:482] T 00000000000000000000000000000000 P fef1f5fd89df475785d1581e53b22d43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:42:50.013886 27268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fef1f5fd89df475785d1581e53b22d43 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:50.014935 27268 raft_consensus.cc:504] T 00000000000000000000000000000000 P fef1f5fd89df475785d1581e53b22d43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef1f5fd89df475785d1581e53b22d43" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 43007 } }
I20260619 18:42:50.015372 27268 leader_election.cc:302] T 00000000000000000000000000000000 P fef1f5fd89df475785d1581e53b22d43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fef1f5fd89df475785d1581e53b22d43; no voters:
I20260619 18:42:50.015731 27268 leader_election.cc:288] T 00000000000000000000000000000000 P fef1f5fd89df475785d1581e53b22d43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:42:50.015853 27273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fef1f5fd89df475785d1581e53b22d43 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:50.016103 27273 raft_consensus.cc:686] T 00000000000000000000000000000000 P fef1f5fd89df475785d1581e53b22d43 [term 1 LEADER]: Becoming Leader. State: Replica: fef1f5fd89df475785d1581e53b22d43, State: Running, Role: LEADER
I20260619 18:42:50.016412 27273 consensus_queue.cc:237] T 00000000000000000000000000000000 P fef1f5fd89df475785d1581e53b22d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef1f5fd89df475785d1581e53b22d43" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 43007 } }
I20260619 18:42:50.016968 27268 sys_catalog.cc:564] T 00000000000000000000000000000000 P fef1f5fd89df475785d1581e53b22d43 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:42:50.018020 27275 sys_catalog.cc:455] T 00000000000000000000000000000000 P fef1f5fd89df475785d1581e53b22d43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fef1f5fd89df475785d1581e53b22d43. Latest consensus state: current_term: 1 leader_uuid: "fef1f5fd89df475785d1581e53b22d43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef1f5fd89df475785d1581e53b22d43" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 43007 } } }
I20260619 18:42:50.018153 27275 sys_catalog.cc:458] T 00000000000000000000000000000000 P fef1f5fd89df475785d1581e53b22d43 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:50.018410 27274 sys_catalog.cc:455] T 00000000000000000000000000000000 P fef1f5fd89df475785d1581e53b22d43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fef1f5fd89df475785d1581e53b22d43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef1f5fd89df475785d1581e53b22d43" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 43007 } } }
I20260619 18:42:50.018508 27274 sys_catalog.cc:458] T 00000000000000000000000000000000 P fef1f5fd89df475785d1581e53b22d43 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:50.018519 27282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:42:50.021206 27282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:42:50.026710 27282 catalog_manager.cc:1347] Generated new cluster ID: bae8c63c1c0c4a7aac80d71f39d2d608
I20260619 18:42:50.026788 27282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:42:50.045933 27282 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:42:50.047276 27282 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:42:50.058111 27282 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fef1f5fd89df475785d1581e53b22d43: Generated new TSK 0
I20260619 18:42:50.058956 27282 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 18:42:50.089612 27298 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 18:42:50.089744 27298 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 18:42:50.090279 27271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:50.073032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47616 (local address 127.21.143.126:43007)
0619 18:42:50.073191 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:50.073196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:50.073476 (+ 280us) server_negotiation.cc:407] Connection header received
0619 18:42:50.073687 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:50.073690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:50.073743 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:50.073833 (+ 90us) server_negotiation.cc:226] Negotiated authn=JWT
0619 18:42:50.074164 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:50.074707 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:50.075964 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:50.076197 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:50.076278 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:50.076667 (+ 389us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 18:42:50.089904 (+ 13237us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33661/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 18:42:50.090159 (+ 255us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47616: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33661/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":65}
W20260619 18:42:50.090387 27271 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47616: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33661/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 18:42:50.091823 27297 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:50.072952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:43007 (local address 127.0.0.1:47616)
0619 18:42:50.073364 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:50.073393 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:50.073528 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:50.073905 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:50.073909 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:50.073918 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0619 18:42:50.074044 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:50.074051 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:50.074813 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:50.074817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:50.075845 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:50.075856 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:50.076317 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:50.076320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:50.076409 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:50.076465 (+ 56us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 18:42:50.090159 (+ 13694us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33661/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 18:42:50.090423 (+ 264us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.143.126:43007: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33661/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":158,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.21.143.126:43007: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33661/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 18:42:50.092567 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 27201
2026-06-19T18:42:50Z chronyd exiting
I20260619 18:42:50.131579 22077 test_util.cc:175] -----------------------------------------------
I20260619 18:42:50.131687 22077 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894542572984-22077-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (331 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 18:42:50.132741 22077 mini_oidc.cc:152] Starting JWKS server
I20260619 18:42:50.133620 22077 webserver.cc:460] Webserver started at https://127.0.0.1:44209/ using document root <none> and password file <none>
I20260619 18:42:50.133752 22077 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 18:42:50.134064 22077 webserver.cc:460] Webserver started at http://127.0.0.1:45837/ using document root <none> and password file <none>
2026-06-19T18:42:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:42:50Z Disabled control of system clock
I20260619 18:42:50.142226 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:39725
--webserver_interface=127.21.143.126
--webserver_port=0
--builtin_ntp_servers=127.21.143.84:33423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:39725
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44209/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894542572984-22077-0/testchainca.cert with env {}
W20260619 18:42:50.252615 27304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:50.252914 27304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:50.253005 27304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:50.257102 27304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:42:50.257192 27304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:50.257221 27304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:50.257261 27304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:42:50.257296 27304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 18:42:50.257334 27304 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 18:42:50.257368 27304 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44209/jwks.json
I20260619 18:42:50.262959 27304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:33423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:39725
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:39725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44209/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894542572984-22077-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:50.264166 27304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:50.265242 27304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:50.274102 27312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:50.274135 27310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:50.274102 27309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:50.274215 27304 server_base.cc:1034] running on GCE node
I20260619 18:42:50.274983 27304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:50.276180 27304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:50.277370 27304 hybrid_clock.cc:648] HybridClock initialized: now 1781894570277351 us; error 30 us; skew 500 ppm
I20260619 18:42:50.279217 27304 webserver.cc:460] Webserver started at http://127.21.143.126:44069/ using document root <none> and password file <none>
I20260619 18:42:50.279876 27304 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:50.279970 27304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:50.280194 27304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:50.282270 27304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "30d13b4437634b74a82dce12c85799a2"
format_stamp: "Formatted at 2026-06-19 18:42:50 on dist-test-slave-mld5"
server_key: "606f6f021a3c72997f662e5fe7275365"
server_key_iv: "e2ea88d371a5eed35c0ae6f9166c1acb"
server_key_version: "encryptionkey@0"
I20260619 18:42:50.282823 27304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "30d13b4437634b74a82dce12c85799a2"
format_stamp: "Formatted at 2026-06-19 18:42:50 on dist-test-slave-mld5"
server_key: "606f6f021a3c72997f662e5fe7275365"
server_key_iv: "e2ea88d371a5eed35c0ae6f9166c1acb"
server_key_version: "encryptionkey@0"
I20260619 18:42:50.286329 27304 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 18:42:50.288775 27318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:50.290068 27304 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:42:50.290223 27304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "30d13b4437634b74a82dce12c85799a2"
format_stamp: "Formatted at 2026-06-19 18:42:50 on dist-test-slave-mld5"
server_key: "606f6f021a3c72997f662e5fe7275365"
server_key_iv: "e2ea88d371a5eed35c0ae6f9166c1acb"
server_key_version: "encryptionkey@0"
I20260619 18:42:50.290340 27304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:50.301627 27304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:50.302248 27304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:50.302438 27304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:50.313516 27304 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:39725
I20260619 18:42:50.313529 27370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:39725 every 8 connection(s)
I20260619 18:42:50.314540 27304 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:42:50.317862 27371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:50.318478 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 27304
I20260619 18:42:50.318629 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:42:50.318925 22077 external_mini_cluster.cc:1428] Setting key 4a454528301658b3554c0475cd0d794f
I20260619 18:42:50.323952 27371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30d13b4437634b74a82dce12c85799a2: Bootstrap starting.
I20260619 18:42:50.326602 27371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30d13b4437634b74a82dce12c85799a2: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:50.327464 27371 log.cc:824] T 00000000000000000000000000000000 P 30d13b4437634b74a82dce12c85799a2: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:50.328442 27374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:50.320412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45496 (local address 127.21.143.126:39725)
0619 18:42:50.321035 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:50.321051 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:50.321087 (+ 36us) server_negotiation.cc:407] Connection header received
0619 18:42:50.321739 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:50.321757 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:50.322154 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:50.322439 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:50.322780 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:50.323807 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:50.324995 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:50.325458 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:50.325574 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:50.326431 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:50.326436 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:50.326462 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:42:50.326895 (+ 433us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:50.326970 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:42:50.326996 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:50.327076 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:50.327230 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 18:42:50.327558 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":137,"threads_started":1}
I20260619 18:42:50.329514 27371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30d13b4437634b74a82dce12c85799a2: No bootstrap required, opened a new log
I20260619 18:42:50.331384 22077 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:42:50.333416 27371 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30d13b4437634b74a82dce12c85799a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d13b4437634b74a82dce12c85799a2" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39725 } }
I20260619 18:42:50.333675 27371 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30d13b4437634b74a82dce12c85799a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:50.333752 27371 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30d13b4437634b74a82dce12c85799a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30d13b4437634b74a82dce12c85799a2, State: Initialized, Role: FOLLOWER
I20260619 18:42:50.334172 27371 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30d13b4437634b74a82dce12c85799a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d13b4437634b74a82dce12c85799a2" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39725 } }
I20260619 18:42:50.334365 27371 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30d13b4437634b74a82dce12c85799a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:42:50.334439 27371 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30d13b4437634b74a82dce12c85799a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:42:50.334532 27371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30d13b4437634b74a82dce12c85799a2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:50.335554 27371 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30d13b4437634b74a82dce12c85799a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d13b4437634b74a82dce12c85799a2" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39725 } }
I20260619 18:42:50.335891 27371 leader_election.cc:302] T 00000000000000000000000000000000 P 30d13b4437634b74a82dce12c85799a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30d13b4437634b74a82dce12c85799a2; no voters:
I20260619 18:42:50.336189 27371 leader_election.cc:288] T 00000000000000000000000000000000 P 30d13b4437634b74a82dce12c85799a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:42:50.336321 27376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30d13b4437634b74a82dce12c85799a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:50.336578 27376 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30d13b4437634b74a82dce12c85799a2 [term 1 LEADER]: Becoming Leader. State: Replica: 30d13b4437634b74a82dce12c85799a2, State: Running, Role: LEADER
I20260619 18:42:50.336879 27376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30d13b4437634b74a82dce12c85799a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d13b4437634b74a82dce12c85799a2" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39725 } }
I20260619 18:42:50.337410 27371 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30d13b4437634b74a82dce12c85799a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:42:50.338907 27378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30d13b4437634b74a82dce12c85799a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30d13b4437634b74a82dce12c85799a2. Latest consensus state: current_term: 1 leader_uuid: "30d13b4437634b74a82dce12c85799a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d13b4437634b74a82dce12c85799a2" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39725 } } }
I20260619 18:42:50.338912 27377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30d13b4437634b74a82dce12c85799a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30d13b4437634b74a82dce12c85799a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d13b4437634b74a82dce12c85799a2" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39725 } } }
I20260619 18:42:50.339051 27378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30d13b4437634b74a82dce12c85799a2 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:50.339088 27377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30d13b4437634b74a82dce12c85799a2 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:50.339811 27385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:42:50.342738 27385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:42:50.348194 27385 catalog_manager.cc:1347] Generated new cluster ID: 4144e485166d4f46a71a855bf1a59e1f
I20260619 18:42:50.348268 27385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:42:50.364800 27385 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:42:50.365988 27385 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:42:50.371713 27385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30d13b4437634b74a82dce12c85799a2: Generated new TSK 0
I20260619 18:42:50.372483 27385 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 18:42:50.400404 27401 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 18:42:50.400533 27401 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 18:42:50.401285 27374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:50.384919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45498 (local address 127.21.143.126:39725)
0619 18:42:50.385064 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:50.385069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:50.385233 (+ 164us) server_negotiation.cc:407] Connection header received
0619 18:42:50.385377 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:50.385380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:50.385457 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:50.385574 (+ 117us) server_negotiation.cc:226] Negotiated authn=JWT
0619 18:42:50.385833 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:50.386689 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:50.387637 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:50.387991 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:50.388097 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:50.388389 (+ 292us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 18:42:50.400823 (+ 12434us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44209/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 18:42:50.401123 (+ 300us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45498: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44209/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 18:42:50.401360 27374 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45498: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44209/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 18:42:50.401461 27400 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:50.384806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39725 (local address 127.0.0.1:45498)
0619 18:42:50.385144 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:50.385158 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:50.385261 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:50.385604 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:50.385607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:50.385614 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0619 18:42:50.385723 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:50.385728 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:50.386800 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:50.386803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:50.387530 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:50.387539 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:50.388113 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:50.388116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:50.388187 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:50.388210 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 18:42:50.401064 (+ 12854us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44209/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 18:42:50.401243 (+ 179us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.143.126:39725: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44209/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":129,"threads_started":1}
I20260619 18:42:50.402361 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 27304
2026-06-19T18:42:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (309 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 18:42:50.441903 22077 mini_oidc.cc:152] Starting JWKS server
I20260619 18:42:50.442770 22077 webserver.cc:460] Webserver started at https://127.0.0.1:37787/ using document root <none> and password file <none>
I20260619 18:42:50.442884 22077 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 18:42:50.443187 22077 webserver.cc:460] Webserver started at http://127.0.0.1:46157/ using document root <none> and password file <none>
2026-06-19T18:42:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:42:50Z Disabled control of system clock
I20260619 18:42:50.451673 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:42527
--webserver_interface=127.21.143.126
--webserver_port=0
--builtin_ntp_servers=127.21.143.84:34939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:42527
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37787/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 18:42:50.557746 27407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:50.558000 27407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:50.558039 27407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:50.562125 27407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:42:50.562187 27407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:50.562203 27407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:50.562254 27407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:42:50.562294 27407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 18:42:50.562322 27407 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 18:42:50.562343 27407 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37787/jwks.json
I20260619 18:42:50.568797 27407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:34939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:42527
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:42527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37787/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:50.570210 27407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:50.571738 27407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:50.580459 27412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:50.580447 27415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:50.580623 27407 server_base.cc:1034] running on GCE node
W20260619 18:42:50.580636 27413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:50.581329 27407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:50.582536 27407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:50.583726 27407 hybrid_clock.cc:648] HybridClock initialized: now 1781894570583714 us; error 24 us; skew 500 ppm
I20260619 18:42:50.585701 27407 webserver.cc:460] Webserver started at http://127.21.143.126:38197/ using document root <none> and password file <none>
I20260619 18:42:50.586357 27407 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:50.586452 27407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:50.586690 27407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:50.588855 27407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "369db0de2b1d4a99ab00ce388f3425f7"
format_stamp: "Formatted at 2026-06-19 18:42:50 on dist-test-slave-mld5"
server_key: "e5c9d162b56412c54638bc3acbf4b3aa"
server_key_iv: "96d7e67efb3c195684f11394938524eb"
server_key_version: "encryptionkey@0"
I20260619 18:42:50.589429 27407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "369db0de2b1d4a99ab00ce388f3425f7"
format_stamp: "Formatted at 2026-06-19 18:42:50 on dist-test-slave-mld5"
server_key: "e5c9d162b56412c54638bc3acbf4b3aa"
server_key_iv: "96d7e67efb3c195684f11394938524eb"
server_key_version: "encryptionkey@0"
I20260619 18:42:50.593087 27407 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:42:50.595481 27421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:50.596709 27407 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:42:50.596848 27407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "369db0de2b1d4a99ab00ce388f3425f7"
format_stamp: "Formatted at 2026-06-19 18:42:50 on dist-test-slave-mld5"
server_key: "e5c9d162b56412c54638bc3acbf4b3aa"
server_key_iv: "96d7e67efb3c195684f11394938524eb"
server_key_version: "encryptionkey@0"
I20260619 18:42:50.596959 27407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:50.622084 27407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:50.622813 27407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:50.623029 27407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:50.629693 27407 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:42527
I20260619 18:42:50.629709 27473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:42527 every 8 connection(s)
I20260619 18:42:50.630659 27407 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:42:50.633455 27474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:50.637014 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 27407
I20260619 18:42:50.637173 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:42:50.637440 22077 external_mini_cluster.cc:1428] Setting key cfe3fb489f4e38ef6c129610e1de9980
I20260619 18:42:50.639169 27474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 369db0de2b1d4a99ab00ce388f3425f7: Bootstrap starting.
I20260619 18:42:50.641403 27474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 369db0de2b1d4a99ab00ce388f3425f7: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:50.642156 27474 log.cc:824] T 00000000000000000000000000000000 P 369db0de2b1d4a99ab00ce388f3425f7: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:50.644246 27474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 369db0de2b1d4a99ab00ce388f3425f7: No bootstrap required, opened a new log
I20260619 18:42:50.646080 27477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:50.638672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54998 (local address 127.21.143.126:42527)
0619 18:42:50.639272 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:50.639332 (+ 60us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:50.639378 (+ 46us) server_negotiation.cc:407] Connection header received
0619 18:42:50.639960 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:50.639975 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:50.640347 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:50.640614 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:50.640931 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:50.641946 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:50.643050 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:50.643580 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:50.643697 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:50.644480 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:50.644485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:50.644497 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:42:50.644853 (+ 356us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:50.644904 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:42:50.644922 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:50.644994 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:50.645156 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 18:42:50.645421 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":118,"threads_started":1}
I20260619 18:42:50.648006 27474 raft_consensus.cc:348] T 00000000000000000000000000000000 P 369db0de2b1d4a99ab00ce388f3425f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "369db0de2b1d4a99ab00ce388f3425f7" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42527 } }
I20260619 18:42:50.648236 27474 raft_consensus.cc:374] T 00000000000000000000000000000000 P 369db0de2b1d4a99ab00ce388f3425f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:50.648304 27474 raft_consensus.cc:729] T 00000000000000000000000000000000 P 369db0de2b1d4a99ab00ce388f3425f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 369db0de2b1d4a99ab00ce388f3425f7, State: Initialized, Role: FOLLOWER
I20260619 18:42:50.648785 27474 consensus_queue.cc:260] T 00000000000000000000000000000000 P 369db0de2b1d4a99ab00ce388f3425f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "369db0de2b1d4a99ab00ce388f3425f7" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42527 } }
I20260619 18:42:50.648909 22077 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:42:50.648938 27474 raft_consensus.cc:388] T 00000000000000000000000000000000 P 369db0de2b1d4a99ab00ce388f3425f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:42:50.649017 27474 raft_consensus.cc:482] T 00000000000000000000000000000000 P 369db0de2b1d4a99ab00ce388f3425f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:42:50.649115 27474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 369db0de2b1d4a99ab00ce388f3425f7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:50.650043 27474 raft_consensus.cc:504] T 00000000000000000000000000000000 P 369db0de2b1d4a99ab00ce388f3425f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "369db0de2b1d4a99ab00ce388f3425f7" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42527 } }
I20260619 18:42:50.650377 27474 leader_election.cc:302] T 00000000000000000000000000000000 P 369db0de2b1d4a99ab00ce388f3425f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 369db0de2b1d4a99ab00ce388f3425f7; no voters:
I20260619 18:42:50.650720 27474 leader_election.cc:288] T 00000000000000000000000000000000 P 369db0de2b1d4a99ab00ce388f3425f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:42:50.650810 27479 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 369db0de2b1d4a99ab00ce388f3425f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:50.651028 27479 raft_consensus.cc:686] T 00000000000000000000000000000000 P 369db0de2b1d4a99ab00ce388f3425f7 [term 1 LEADER]: Becoming Leader. State: Replica: 369db0de2b1d4a99ab00ce388f3425f7, State: Running, Role: LEADER
I20260619 18:42:50.651345 27479 consensus_queue.cc:237] T 00000000000000000000000000000000 P 369db0de2b1d4a99ab00ce388f3425f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "369db0de2b1d4a99ab00ce388f3425f7" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42527 } }
I20260619 18:42:50.651903 27474 sys_catalog.cc:564] T 00000000000000000000000000000000 P 369db0de2b1d4a99ab00ce388f3425f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:42:50.652958 27481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 369db0de2b1d4a99ab00ce388f3425f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 369db0de2b1d4a99ab00ce388f3425f7. Latest consensus state: current_term: 1 leader_uuid: "369db0de2b1d4a99ab00ce388f3425f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "369db0de2b1d4a99ab00ce388f3425f7" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42527 } } }
I20260619 18:42:50.652957 27480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 369db0de2b1d4a99ab00ce388f3425f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "369db0de2b1d4a99ab00ce388f3425f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "369db0de2b1d4a99ab00ce388f3425f7" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42527 } } }
I20260619 18:42:50.653112 27481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 369db0de2b1d4a99ab00ce388f3425f7 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:50.653112 27480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 369db0de2b1d4a99ab00ce388f3425f7 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:50.653548 27488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:42:50.656289 27488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:42:50.661896 27488 catalog_manager.cc:1347] Generated new cluster ID: 7af7e532545d4a6c94eaa1de899915a1
I20260619 18:42:50.661994 27488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:42:50.681216 27488 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:42:50.682220 27488 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:42:50.686925 27488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 369db0de2b1d4a99ab00ce388f3425f7: Generated new TSK 0
I20260619 18:42:50.687618 27488 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:42:50.700083 27477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:50.695368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55000 (local address 127.21.143.126:42527)
0619 18:42:50.695502 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:50.695507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:50.695724 (+ 217us) server_negotiation.cc:407] Connection header received
0619 18:42:50.695870 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:50.695872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:50.695915 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:50.695995 (+ 80us) server_negotiation.cc:226] Negotiated authn=JWT
0619 18:42:50.696241 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:50.696752 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:50.697949 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:50.698159 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:50.698226 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:50.698534 (+ 308us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 18:42:50.699749 (+ 1215us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37787/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 18:42:50.699972 (+ 223us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55000: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37787/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":45}
W20260619 18:42:50.700158 27477 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55000: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37787/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 18:42:50.703016 27503 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:50.695285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42527 (local address 127.0.0.1:55000)
0619 18:42:50.695637 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:50.695651 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:50.695760 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:50.696017 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:50.696020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:50.696028 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 18:42:50.696141 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:50.696147 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:50.696859 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:50.696863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:50.697835 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:50.697845 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:50.698266 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:50.698270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:50.698357 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:50.698383 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 18:42:50.699904 (+ 1521us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37787/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 18:42:50.702887 (+ 2983us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.143.126:42527: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37787/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":2821,"thread_start_us":116,"threads_started":1}
I20260619 18:42:50.703572 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 27407
E20260619 18:42:50.712214 27504 webserver.cc:533] Webserver: sslize failed:
E20260619 18:42:50.712273 27504 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-19T18:42:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (300 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T18:42:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:42:50Z Disabled control of system clock
I20260619 18:42:50.751380 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:38351
--webserver_interface=127.21.143.126
--webserver_port=0
--builtin_ntp_servers=127.21.143.84:44487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:38351
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894542572984-22077-0/testchainca.cert with env {}
W20260619 18:42:50.861176 27508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:50.861541 27508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:50.861637 27508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:50.865805 27508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:42:50.865896 27508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:50.865924 27508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:50.865963 27508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:42:50.866003 27508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 18:42:50.866063 27508 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 18:42:50.866096 27508 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 18:42:50.871762 27508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:44487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:38351
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:38351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894542572984-22077-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:50.872939 27508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:50.874034 27508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:50.883075 27513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:50.883031 27514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:50.883117 27508 server_base.cc:1034] running on GCE node
W20260619 18:42:50.883038 27516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:50.883883 27508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:50.885018 27508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:50.886179 27508 hybrid_clock.cc:648] HybridClock initialized: now 1781894570886165 us; error 29 us; skew 500 ppm
I20260619 18:42:50.888087 27508 webserver.cc:460] Webserver started at http://127.21.143.126:33145/ using document root <none> and password file <none>
I20260619 18:42:50.888686 27508 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:50.888749 27508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:50.888981 27508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:50.891034 27508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "950b3b971d3f4151b34f120b7a81266c"
format_stamp: "Formatted at 2026-06-19 18:42:50 on dist-test-slave-mld5"
server_key: "4e7119a32b9e8af9644749a94562f86f"
server_key_iv: "31444faf58f1a061081bede6295f6036"
server_key_version: "encryptionkey@0"
I20260619 18:42:50.891662 27508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "950b3b971d3f4151b34f120b7a81266c"
format_stamp: "Formatted at 2026-06-19 18:42:50 on dist-test-slave-mld5"
server_key: "4e7119a32b9e8af9644749a94562f86f"
server_key_iv: "31444faf58f1a061081bede6295f6036"
server_key_version: "encryptionkey@0"
I20260619 18:42:50.895409 27508 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:42:50.897842 27522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:50.899083 27508 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:42:50.899225 27508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "950b3b971d3f4151b34f120b7a81266c"
format_stamp: "Formatted at 2026-06-19 18:42:50 on dist-test-slave-mld5"
server_key: "4e7119a32b9e8af9644749a94562f86f"
server_key_iv: "31444faf58f1a061081bede6295f6036"
server_key_version: "encryptionkey@0"
I20260619 18:42:50.899362 27508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:50.913564 27508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:50.914279 27508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:50.914506 27508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:50.921528 27508 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:38351
I20260619 18:42:50.921535 27574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:38351 every 8 connection(s)
I20260619 18:42:50.922541 27508 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:42:50.925451 27575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:50.926789 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 27508
I20260619 18:42:50.926923 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:42:50.927259 22077 external_mini_cluster.cc:1428] Setting key 645b338901b4a0d34e6d63836f48d245
I20260619 18:42:50.931766 27575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 950b3b971d3f4151b34f120b7a81266c: Bootstrap starting.
I20260619 18:42:50.933948 27575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 950b3b971d3f4151b34f120b7a81266c: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:50.934782 27575 log.cc:824] T 00000000000000000000000000000000 P 950b3b971d3f4151b34f120b7a81266c: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:50.936444 27578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:50.928711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35450 (local address 127.21.143.126:38351)
0619 18:42:50.929359 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:50.929375 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:50.929410 (+ 35us) server_negotiation.cc:407] Connection header received
0619 18:42:50.930064 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:50.930082 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:50.930479 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:50.930812 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:50.931247 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:50.932357 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:50.933222 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:50.933685 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:50.933852 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:50.934680 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:50.934685 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:50.934698 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:42:50.935128 (+ 430us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:50.935189 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:42:50.935209 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:50.935332 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:50.935484 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 18:42:50.935734 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":134,"threads_started":1}
I20260619 18:42:50.936803 27575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 950b3b971d3f4151b34f120b7a81266c: No bootstrap required, opened a new log
I20260619 18:42:50.939100 22077 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:42:50.940598 27575 raft_consensus.cc:348] T 00000000000000000000000000000000 P 950b3b971d3f4151b34f120b7a81266c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950b3b971d3f4151b34f120b7a81266c" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 38351 } }
I20260619 18:42:50.940881 27575 raft_consensus.cc:374] T 00000000000000000000000000000000 P 950b3b971d3f4151b34f120b7a81266c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:50.940960 27575 raft_consensus.cc:729] T 00000000000000000000000000000000 P 950b3b971d3f4151b34f120b7a81266c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 950b3b971d3f4151b34f120b7a81266c, State: Initialized, Role: FOLLOWER
I20260619 18:42:50.941457 27575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 950b3b971d3f4151b34f120b7a81266c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950b3b971d3f4151b34f120b7a81266c" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 38351 } }
I20260619 18:42:50.941605 27575 raft_consensus.cc:388] T 00000000000000000000000000000000 P 950b3b971d3f4151b34f120b7a81266c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:42:50.941673 27575 raft_consensus.cc:482] T 00000000000000000000000000000000 P 950b3b971d3f4151b34f120b7a81266c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:42:50.941768 27575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 950b3b971d3f4151b34f120b7a81266c [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:50.942621 27575 raft_consensus.cc:504] T 00000000000000000000000000000000 P 950b3b971d3f4151b34f120b7a81266c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950b3b971d3f4151b34f120b7a81266c" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 38351 } }
I20260619 18:42:50.942967 27575 leader_election.cc:302] T 00000000000000000000000000000000 P 950b3b971d3f4151b34f120b7a81266c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 950b3b971d3f4151b34f120b7a81266c; no voters:
I20260619 18:42:50.943270 27575 leader_election.cc:288] T 00000000000000000000000000000000 P 950b3b971d3f4151b34f120b7a81266c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:42:50.943459 27580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 950b3b971d3f4151b34f120b7a81266c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:50.943701 27580 raft_consensus.cc:686] T 00000000000000000000000000000000 P 950b3b971d3f4151b34f120b7a81266c [term 1 LEADER]: Becoming Leader. State: Replica: 950b3b971d3f4151b34f120b7a81266c, State: Running, Role: LEADER
I20260619 18:42:50.944000 27580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 950b3b971d3f4151b34f120b7a81266c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950b3b971d3f4151b34f120b7a81266c" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 38351 } }
I20260619 18:42:50.944519 27575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 950b3b971d3f4151b34f120b7a81266c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:42:50.945794 27582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 950b3b971d3f4151b34f120b7a81266c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 950b3b971d3f4151b34f120b7a81266c. Latest consensus state: current_term: 1 leader_uuid: "950b3b971d3f4151b34f120b7a81266c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950b3b971d3f4151b34f120b7a81266c" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 38351 } } }
I20260619 18:42:50.945948 27582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 950b3b971d3f4151b34f120b7a81266c [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:50.945801 27581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 950b3b971d3f4151b34f120b7a81266c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "950b3b971d3f4151b34f120b7a81266c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950b3b971d3f4151b34f120b7a81266c" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 38351 } } }
I20260619 18:42:50.946033 27581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 950b3b971d3f4151b34f120b7a81266c [sys.catalog]: This master's current role is: LEADER
W20260619 18:42:50.949329 27596 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 950b3b971d3f4151b34f120b7a81266c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:42:50.949396 27596 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:42:50.949479 27590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:42:50.950160 27590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:42:50.955206 27590 catalog_manager.cc:1347] Generated new cluster ID: 190e427db1e34c03aefa1ae3b79675c7
I20260619 18:42:50.955277 27590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:42:50.971150 27590 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:42:50.972163 27590 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:42:50.978821 27590 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 950b3b971d3f4151b34f120b7a81266c: Generated new TSK 0
I20260619 18:42:50.979708 27590 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:42:51.014061 27578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:50.993384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35458 (local address 127.21.143.126:38351)
0619 18:42:50.993512 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:50.993518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:50.993692 (+ 174us) server_negotiation.cc:407] Connection header received
0619 18:42:50.993898 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:50.993900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:50.993949 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:50.994063 (+ 114us) server_negotiation.cc:226] Negotiated authn=JWT
0619 18:42:50.994306 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:50.994855 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:50.995831 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:50.996056 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:50.996184 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:50.996499 (+ 315us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 18:42:51.013590 (+ 17091us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 18:42:51.013903 (+ 313us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35458: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":44}
W20260619 18:42:51.014143 27578 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35458: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 18:42:51.014192 27604 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:50.993284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:38351 (local address 127.0.0.1:35458)
0619 18:42:50.993604 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:50.993617 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:50.993743 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:50.994071 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:50.994074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:50.994082 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 18:42:50.994193 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:50.994205 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:50.994968 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:50.994972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:50.995704 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:50.995712 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:50.996199 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:50.996202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:50.996303 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:50.996324 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 18:42:51.013823 (+ 17499us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 18:42:51.013990 (+ 167us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.143.126:38351: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":102,"threads_started":1}
I20260619 18:42:51.015101 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 27508
2026-06-19T18:42:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (290 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (123 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (125 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:42:51 dist-test-slave-mld5 krb5kdc[27616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:42:51 dist-test-slave-mld5 krb5kdc[27616](info): set up 2 sockets
Jun 19 18:42:51 dist-test-slave-mld5 krb5kdc[27616](info): commencing operation
krb5kdc: starting...
W20260619 18:42:52.311784 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:42:52 dist-test-slave-mld5 krb5kdc[27616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:42:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:42:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.143.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
I20260619 18:42:52.433943 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:39541
--webserver_interface=127.21.143.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:45951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:39541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:52.543390 27629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:52.543681 27629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:52.543722 27629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:52.547780 27629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:42:52.547834 27629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:52.547853 27629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:52.547916 27629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:42:52.547950 27629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:42:52.553345 27629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:45951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:39541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:39541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:52.554387 27629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:52.555403 27629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:52.565646 27635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:52.565670 27637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:52.565665 27634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:52.566247 27629 server_base.cc:1034] running on GCE node
I20260619 18:42:52.566946 27629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:52.568140 27629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:52.569474 27629 hybrid_clock.cc:648] HybridClock initialized: now 1781894572569465 us; error 32 us; skew 500 ppm
Jun 19 18:42:52 dist-test-slave-mld5 krb5kdc[27616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894572, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:52.572480 27629 init.cc:377] Logged in from keytab as kudu/127.21.143.126@KRBTEST.COM (short username kudu)
I20260619 18:42:52.573391 27629 webserver.cc:460] Webserver started at http://127.21.143.126:33877/ using document root <none> and password file <none>
I20260619 18:42:52.573984 27629 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:52.574079 27629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:52.574322 27629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:52.576484 27629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "59f151d4c15b4141a66d99369416d4ca"
format_stamp: "Formatted at 2026-06-19 18:42:52 on dist-test-slave-mld5"
server_key: "e6cd7f4fe0088d6c3c9d0aa523f03b75"
server_key_iv: "7c2041be33127056de66c515179c9ced"
server_key_version: "encryptionkey@0"
I20260619 18:42:52.577066 27629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "59f151d4c15b4141a66d99369416d4ca"
format_stamp: "Formatted at 2026-06-19 18:42:52 on dist-test-slave-mld5"
server_key: "e6cd7f4fe0088d6c3c9d0aa523f03b75"
server_key_iv: "7c2041be33127056de66c515179c9ced"
server_key_version: "encryptionkey@0"
I20260619 18:42:52.580755 27629 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:42:52.583091 27644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:52.584383 27629 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:42:52.584522 27629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "59f151d4c15b4141a66d99369416d4ca"
format_stamp: "Formatted at 2026-06-19 18:42:52 on dist-test-slave-mld5"
server_key: "e6cd7f4fe0088d6c3c9d0aa523f03b75"
server_key_iv: "7c2041be33127056de66c515179c9ced"
server_key_version: "encryptionkey@0"
I20260619 18:42:52.584635 27629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:52.604628 27629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:52.608112 27629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:52.608338 27629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:52.615587 27629 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:39541
I20260619 18:42:52.615614 27696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:39541 every 8 connection(s)
I20260619 18:42:52.616600 27629 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:42:52.619398 27697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:52.620119 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 27629
I20260619 18:42:52.620222 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:42:52.620548 22077 external_mini_cluster.cc:1428] Setting key cce75565ca22a74616b7208f09da115f
I20260619 18:42:52.625844 27697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59f151d4c15b4141a66d99369416d4ca: Bootstrap starting.
I20260619 18:42:52.628201 27697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59f151d4c15b4141a66d99369416d4ca: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:52.628896 27697 log.cc:824] T 00000000000000000000000000000000 P 59f151d4c15b4141a66d99369416d4ca: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:42:52 dist-test-slave-mld5 krb5kdc[27616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:52.630890 27697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59f151d4c15b4141a66d99369416d4ca: No bootstrap required, opened a new log
I20260619 18:42:52.635280 27700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:52.621906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51214 (local address 127.21.143.126:39541)
0619 18:42:52.622313 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:52.622331 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:52.622373 (+ 42us) server_negotiation.cc:407] Connection header received
0619 18:42:52.623181 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:52.623204 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:52.623635 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:52.624006 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:52.624650 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:52.626043 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:52.626866 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:52.627575 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:52.627731 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:52.630284 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:52.630290 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:52.630305 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:52.630333 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:52.632326 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:52.632749 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:52.632755 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:52.632760 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:52.632910 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:52.633164 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:52.633168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:52.633171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:52.633669 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:52.633845 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:52.634124 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 18:42:52.634405 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":87,"threads_started":1}
I20260619 18:42:52.635392 27697 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59f151d4c15b4141a66d99369416d4ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59f151d4c15b4141a66d99369416d4ca" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39541 } }
I20260619 18:42:52.635635 27697 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59f151d4c15b4141a66d99369416d4ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:52.635715 27697 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59f151d4c15b4141a66d99369416d4ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f151d4c15b4141a66d99369416d4ca, State: Initialized, Role: FOLLOWER
I20260619 18:42:52.636159 27697 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59f151d4c15b4141a66d99369416d4ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59f151d4c15b4141a66d99369416d4ca" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39541 } }
I20260619 18:42:52.636271 27697 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59f151d4c15b4141a66d99369416d4ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:42:52.636355 27697 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59f151d4c15b4141a66d99369416d4ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:42:52.636447 27697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59f151d4c15b4141a66d99369416d4ca [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:52.637403 27697 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59f151d4c15b4141a66d99369416d4ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59f151d4c15b4141a66d99369416d4ca" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39541 } }
I20260619 18:42:52.637737 27697 leader_election.cc:302] T 00000000000000000000000000000000 P 59f151d4c15b4141a66d99369416d4ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59f151d4c15b4141a66d99369416d4ca; no voters:
I20260619 18:42:52.638019 27697 leader_election.cc:288] T 00000000000000000000000000000000 P 59f151d4c15b4141a66d99369416d4ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:42:52.638130 27702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59f151d4c15b4141a66d99369416d4ca [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:52.638371 27702 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59f151d4c15b4141a66d99369416d4ca [term 1 LEADER]: Becoming Leader. State: Replica: 59f151d4c15b4141a66d99369416d4ca, State: Running, Role: LEADER
I20260619 18:42:52.638634 27702 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59f151d4c15b4141a66d99369416d4ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59f151d4c15b4141a66d99369416d4ca" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39541 } }
I20260619 18:42:52.639264 27697 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59f151d4c15b4141a66d99369416d4ca [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:42:52.640328 27704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59f151d4c15b4141a66d99369416d4ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59f151d4c15b4141a66d99369416d4ca. Latest consensus state: current_term: 1 leader_uuid: "59f151d4c15b4141a66d99369416d4ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59f151d4c15b4141a66d99369416d4ca" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39541 } } }
I20260619 18:42:52.640465 27704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59f151d4c15b4141a66d99369416d4ca [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:52.640758 27703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59f151d4c15b4141a66d99369416d4ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59f151d4c15b4141a66d99369416d4ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59f151d4c15b4141a66d99369416d4ca" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39541 } } }
I20260619 18:42:52.640859 27703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59f151d4c15b4141a66d99369416d4ca [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:52.640816 27711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:42:52.643836 27711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:42:52.649127 27711 catalog_manager.cc:1347] Generated new cluster ID: a15241fe9a764c2dabf9d037ede365f4
I20260619 18:42:52.649186 27711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:42:52.678959 27711 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:42:52.680320 27711 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:42:52.689543 27711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59f151d4c15b4141a66d99369416d4ca: Generated new TSK 0
I20260619 18:42:52.690296 27711 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.143.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab.
Entry for principal kudu/127.21.143.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab.
I20260619 18:42:52.722302 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.65:0
--local_ip_for_outbound_sockets=127.21.143.65
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:39541
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:45951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:52.833709 27723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:52.833967 27723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:52.834010 27723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:52.838171 27723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:52.838232 27723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:52.838325 27723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.65
I20260619 18:42:52.843832 27723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:45951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:39541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.65
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:52.844902 27723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:52.845930 27723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:52.855252 27731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:52.855321 27728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:52.855396 27729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:52.855465 27723 server_base.cc:1034] running on GCE node
I20260619 18:42:52.855953 27723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:52.856608 27723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:52.857769 27723 hybrid_clock.cc:648] HybridClock initialized: now 1781894572857755 us; error 31 us; skew 500 ppm
Jun 19 18:42:52 dist-test-slave-mld5 krb5kdc[27616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894572, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:52.860598 27723 init.cc:377] Logged in from keytab as kudu/127.21.143.65@KRBTEST.COM (short username kudu)
I20260619 18:42:52.861483 27723 webserver.cc:460] Webserver started at http://127.21.143.65:46799/ using document root <none> and password file <none>
I20260619 18:42:52.862010 27723 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:52.862067 27723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:52.862226 27723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:52.864297 27723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-0/data/instance:
uuid: "f61109b7d11f47c9b44a763730e4b445"
format_stamp: "Formatted at 2026-06-19 18:42:52 on dist-test-slave-mld5"
server_key: "237419752f962e0d3a80f0d228fda036"
server_key_iv: "f01b74deef5ca0414fc37f35010101e1"
server_key_version: "encryptionkey@0"
I20260619 18:42:52.864790 27723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance:
uuid: "f61109b7d11f47c9b44a763730e4b445"
format_stamp: "Formatted at 2026-06-19 18:42:52 on dist-test-slave-mld5"
server_key: "237419752f962e0d3a80f0d228fda036"
server_key_iv: "f01b74deef5ca0414fc37f35010101e1"
server_key_version: "encryptionkey@0"
I20260619 18:42:52.868645 27723 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:42:52.871040 27738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:52.872450 27723 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:42:52.872598 27723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-0/wal
uuid: "f61109b7d11f47c9b44a763730e4b445"
format_stamp: "Formatted at 2026-06-19 18:42:52 on dist-test-slave-mld5"
server_key: "237419752f962e0d3a80f0d228fda036"
server_key_iv: "f01b74deef5ca0414fc37f35010101e1"
server_key_version: "encryptionkey@0"
I20260619 18:42:52.872721 27723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:52.887145 27723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:52.890466 27723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:52.890684 27723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:52.891376 27723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:52.892221 27723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:52.892302 27723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:52.892373 27723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:52.892426 27723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:52.901536 27723 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.65:42021
I20260619 18:42:52.901554 27851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.65:42021 every 8 connection(s)
I20260619 18:42:52.902462 27723 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
I20260619 18:42:52.908409 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 27723
I20260619 18:42:52.908541 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance
I20260619 18:42:52.908831 22077 external_mini_cluster.cc:1428] Setting key 095e335f05bc042710aadaf802d78a1c
Jun 19 18:42:52 dist-test-slave-mld5 krb5kdc[27616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894572, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:52.916826 27700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:52.904395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:45555 (local address 127.21.143.126:39541)
0619 18:42:52.904533 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:52.904536 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:52.905357 (+ 821us) server_negotiation.cc:407] Connection header received
0619 18:42:52.906719 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:52.906722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:52.906784 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:52.906886 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:52.908157 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:52.908799 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:52.910050 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:52.910292 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:52.910400 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:52.913126 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:52.913129 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:52.913131 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:52.913155 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:52.914695 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:52.915211 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:52.915214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:52.915216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:52.915350 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:52.915703 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:52.915708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:52.915709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:52.915902 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:52.916003 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:52.916522 (+ 519us) server_negotiation.cc:299] Negotiation successful
0619 18:42:52.916656 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 18:42:52.917677 27854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:52.904736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39541 (local address 127.21.143.65:45555)
0619 18:42:52.905205 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:52.905247 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:52.906495 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:52.907064 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:52.907072 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:52.907514 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:52.907998 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:52.908011 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:52.908934 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:52.908937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:52.909923 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:52.909932 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:52.910436 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:52.910440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:52.910623 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:52.911225 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:52.911240 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:52.912940 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:52.914816 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:52.914822 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:52.914839 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:52.915103 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:52.915453 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:52.915457 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:52.915459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:52.915605 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:52.916015 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:52.916023 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:52.916279 (+ 256us) client_negotiation.cc:769] Sending connection context
0619 18:42:52.916474 (+ 195us) client_negotiation.cc:240] Negotiation successful
0619 18:42:52.916823 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for kudu/127.21.143.66@KRBTEST.COM; defaulting to no policy
I20260619 18:42:52.919095 27852 heartbeater.cc:347] Connected to a master server at 127.21.143.126:39541
I20260619 18:42:52.919380 27852 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:52.919857 27852 heartbeater.cc:511] Master 127.21.143.126:39541 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.66@KRBTEST.COM" created.
I20260619 18:42:52.921689 27661 ts_manager.cc:194] Registered new tserver with Master: f61109b7d11f47c9b44a763730e4b445 (127.21.143.65:42021)
I20260619 18:42:52.922979 27661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.65@KRBTEST.COM'} at 127.21.143.65:45555
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab.
Entry for principal kudu/127.21.143.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab.
I20260619 18:42:52.936272 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.66:0
--local_ip_for_outbound_sockets=127.21.143.66
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:39541
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:45951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:53.044858 27857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:53.045135 27857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:53.045178 27857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:53.049176 27857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:53.049227 27857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:53.049283 27857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.66
I20260619 18:42:53.054672 27857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:45951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:39541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.66
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:53.055748 27857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:53.056735 27857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:53.066260 27863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:53.066277 27862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:53.066246 27865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:53.066368 27857 server_base.cc:1034] running on GCE node
I20260619 18:42:53.067117 27857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:53.067814 27857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:53.068974 27857 hybrid_clock.cc:648] HybridClock initialized: now 1781894573068959 us; error 27 us; skew 500 ppm
Jun 19 18:42:53 dist-test-slave-mld5 krb5kdc[27616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894573, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:53.076072 27857 init.cc:377] Logged in from keytab as kudu/127.21.143.66@KRBTEST.COM (short username kudu)
I20260619 18:42:53.077078 27857 webserver.cc:460] Webserver started at http://127.21.143.66:41193/ using document root <none> and password file <none>
I20260619 18:42:53.077698 27857 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:53.077770 27857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:53.077955 27857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:53.080097 27857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-1/data/instance:
uuid: "3dd01328f47f4638a1bbac94c98336c6"
format_stamp: "Formatted at 2026-06-19 18:42:53 on dist-test-slave-mld5"
server_key: "9ccbf7cd82227cc39e63e9b2399b5d8c"
server_key_iv: "a627b6ed2af690b4e9b5e453352b3edc"
server_key_version: "encryptionkey@0"
I20260619 18:42:53.080618 27857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance:
uuid: "3dd01328f47f4638a1bbac94c98336c6"
format_stamp: "Formatted at 2026-06-19 18:42:53 on dist-test-slave-mld5"
server_key: "9ccbf7cd82227cc39e63e9b2399b5d8c"
server_key_iv: "a627b6ed2af690b4e9b5e453352b3edc"
server_key_version: "encryptionkey@0"
I20260619 18:42:53.084816 27857 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:42:53.087373 27872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:53.088753 27857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:42:53.088899 27857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-1/wal
uuid: "3dd01328f47f4638a1bbac94c98336c6"
format_stamp: "Formatted at 2026-06-19 18:42:53 on dist-test-slave-mld5"
server_key: "9ccbf7cd82227cc39e63e9b2399b5d8c"
server_key_iv: "a627b6ed2af690b4e9b5e453352b3edc"
server_key_version: "encryptionkey@0"
I20260619 18:42:53.089013 27857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:53.110580 27857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:53.113920 27857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:53.114162 27857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:53.114720 27857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:53.115558 27857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:53.115617 27857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:53.115657 27857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:53.115672 27857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:53.124246 27857 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.66:39861
I20260619 18:42:53.124277 27985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.66:39861 every 8 connection(s)
I20260619 18:42:53.125124 27857 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
I20260619 18:42:53.132493 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 27857
I20260619 18:42:53.132616 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance
I20260619 18:42:53.132941 22077 external_mini_cluster.cc:1428] Setting key b6e1dde7a80856e9b449c39813b177a6
Jun 19 18:42:53 dist-test-slave-mld5 krb5kdc[27616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894573, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:53.139493 27700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:53.126888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:46981 (local address 127.21.143.126:39541)
0619 18:42:53.127026 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:53.127030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:53.128064 (+ 1034us) server_negotiation.cc:407] Connection header received
0619 18:42:53.129527 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:53.129530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:53.129603 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:53.129675 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:53.130835 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:53.131438 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:53.132770 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:53.132975 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:53.133087 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:53.135653 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:53.135660 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:53.135665 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:53.135699 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:53.137247 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:53.137784 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:53.137791 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:53.137795 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:53.137912 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:53.138242 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:53.138248 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:53.138253 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:53.138517 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:53.138627 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:53.139121 (+ 494us) server_negotiation.cc:299] Negotiation successful
0619 18:42:53.139273 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 18:42:53.140503 27988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:53.127287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39541 (local address 127.21.143.66:46981)
0619 18:42:53.127889 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:53.127954 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:53.129307 (+ 1353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:53.129888 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:53.129895 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:53.130230 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:53.130699 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:53.130711 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:53.131548 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:53.131552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:53.132623 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:53.132638 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:53.133126 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:53.133130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:53.133312 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:53.133898 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:53.133915 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:53.135497 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:53.137356 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:53.137361 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:53.137374 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:53.137664 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:53.138010 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:53.138012 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:53.138013 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:53.138147 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:53.138621 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:53.138626 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:53.138872 (+ 246us) client_negotiation.cc:769] Sending connection context
0619 18:42:53.139032 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 18:42:53.139436 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":123,"threads_started":1}
I20260619 18:42:53.141794 27986 heartbeater.cc:347] Connected to a master server at 127.21.143.126:39541
I20260619 18:42:53.142033 27986 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:53.142534 27986 heartbeater.cc:511] Master 127.21.143.126:39541 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.143.67@KRBTEST.COM; defaulting to no policy
I20260619 18:42:53.143685 27661 ts_manager.cc:194] Registered new tserver with Master: 3dd01328f47f4638a1bbac94c98336c6 (127.21.143.66:39861)
I20260619 18:42:53.144428 27661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.66@KRBTEST.COM'} at 127.21.143.66:46981
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab.
Entry for principal kudu/127.21.143.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab.
I20260619 18:42:53.162271 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.67:0
--local_ip_for_outbound_sockets=127.21.143.67
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:39541
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:45951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:53.273494 27991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:53.273754 27991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:53.273823 27991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:53.277954 27991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:53.278025 27991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:53.278110 27991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.67
I20260619 18:42:53.282923 27991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:45951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:39541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.67
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:53.283939 27991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:53.284914 27991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 18:42:53.294274 27991 server_base.cc:1034] running on GCE node
W20260619 18:42:53.294250 27999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:53.294257 27997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:53.294416 27996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:53.294886 27991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:53.295517 27991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:53.296705 27991 hybrid_clock.cc:648] HybridClock initialized: now 1781894573296693 us; error 33 us; skew 500 ppm
Jun 19 18:42:53 dist-test-slave-mld5 krb5kdc[27616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894573, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:53.299266 27991 init.cc:377] Logged in from keytab as kudu/127.21.143.67@KRBTEST.COM (short username kudu)
I20260619 18:42:53.300118 27991 webserver.cc:460] Webserver started at http://127.21.143.67:40163/ using document root <none> and password file <none>
I20260619 18:42:53.300668 27991 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:53.300755 27991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:53.300980 27991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:53.303006 27991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-2/data/instance:
uuid: "484112daa8254462b2f742da465dce2d"
format_stamp: "Formatted at 2026-06-19 18:42:53 on dist-test-slave-mld5"
server_key: "ef5a10be4206235a23995b84b5fc9286"
server_key_iv: "0b21ca292d1d528f1e1bdd6caf583356"
server_key_version: "encryptionkey@0"
I20260619 18:42:53.303601 27991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance:
uuid: "484112daa8254462b2f742da465dce2d"
format_stamp: "Formatted at 2026-06-19 18:42:53 on dist-test-slave-mld5"
server_key: "ef5a10be4206235a23995b84b5fc9286"
server_key_iv: "0b21ca292d1d528f1e1bdd6caf583356"
server_key_version: "encryptionkey@0"
I20260619 18:42:53.307281 27991 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 18:42:53.309979 28006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:53.311254 27991 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 18:42:53.311431 27991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-2/wal
uuid: "484112daa8254462b2f742da465dce2d"
format_stamp: "Formatted at 2026-06-19 18:42:53 on dist-test-slave-mld5"
server_key: "ef5a10be4206235a23995b84b5fc9286"
server_key_iv: "0b21ca292d1d528f1e1bdd6caf583356"
server_key_version: "encryptionkey@0"
I20260619 18:42:53.311549 27991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:53.322117 27991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:53.325196 27991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:53.325398 27991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:53.325968 27991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:53.326761 27991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:53.326831 27991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:53.326901 27991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:53.326948 27991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:53.335800 27991 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.67:35225
I20260619 18:42:53.335860 28119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.67:35225 every 8 connection(s)
I20260619 18:42:53.336719 27991 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
I20260619 18:42:53.338423 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 27991
I20260619 18:42:53.338591 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance
I20260619 18:42:53.338917 22077 external_mini_cluster.cc:1428] Setting key c5703a94682c097009b371ae9fd6b8ac
Jun 19 18:42:53 dist-test-slave-mld5 krb5kdc[27616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894573, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:42:53.350531 27700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:53.338509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:53449 (local address 127.21.143.126:39541)
0619 18:42:53.338665 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:53.338670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:53.339710 (+ 1040us) server_negotiation.cc:407] Connection header received
0619 18:42:53.341239 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:53.341242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:53.341317 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:53.341399 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:53.342458 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:53.342972 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:53.344043 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:53.344248 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:53.344329 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:53.346713 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:53.346717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:53.346720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:53.346755 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:53.348383 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:53.348933 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:53.348938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:53.348940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:53.349028 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:53.349398 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:53.349404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:53.349409 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:53.349597 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:53.349711 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:53.350233 (+ 522us) server_negotiation.cc:299] Negotiation successful
0619 18:42:53.350355 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 18:42:53.351477 28122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:53.339036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39541 (local address 127.21.143.67:53449)
0619 18:42:53.339566 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:53.339617 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:53.341023 (+ 1406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:53.341556 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:53.341562 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:53.341880 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:53.342326 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:53.342338 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:53.343073 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:53.343075 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:53.343933 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:53.343942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:53.344369 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:53.344371 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:53.344510 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:53.344960 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:53.344975 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:53.346512 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:53.348528 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:53.348534 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:53.348548 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:53.348819 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:53.349135 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:53.349138 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:53.349140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:53.349285 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:53.349719 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:53.349729 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:53.349990 (+ 261us) client_negotiation.cc:769] Sending connection context
0619 18:42:53.350187 (+ 197us) client_negotiation.cc:240] Negotiation successful
0619 18:42:53.350502 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":173,"threads_started":1}
I20260619 18:42:53.352648 28120 heartbeater.cc:347] Connected to a master server at 127.21.143.126:39541
I20260619 18:42:53.352908 28120 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:53.353394 28120 heartbeater.cc:511] Master 127.21.143.126:39541 requested a full tablet report, sending...
I20260619 18:42:53.354380 27661 ts_manager.cc:194] Registered new tserver with Master: 484112daa8254462b2f742da465dce2d (127.21.143.67:35225)
I20260619 18:42:53.354970 27661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.67@KRBTEST.COM'} at 127.21.143.67:53449
I20260619 18:42:53.364392 22077 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/test-admin.keytab.
Jun 19 18:42:53 dist-test-slave-mld5 krb5kdc[27616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:53.380425 22077 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 18:42:53.383265 28127 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:53.381854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48230 (local address 127.21.143.65:42021)
0619 18:42:53.382159 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:53.382174 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:53.382198 (+ 24us) server_negotiation.cc:407] Connection header received
0619 18:42:53.382249 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:53.382254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:53.382491 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:53.382620 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:53.383079 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48230: BlockingRecv error: recv got EOF from 127.0.0.1:48230 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":106,"threads_started":1}
W20260619 18:42:53.383791 22077 init.cc:244] Error while opening credential cache '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/krb5cc)
Jun 19 18:42:53 dist-test-slave-mld5 krb5kdc[27616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:53.384843 22077 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 18:42:53 dist-test-slave-mld5 krb5kdc[27616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:53.385823 22077 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 18:42:53 dist-test-slave-mld5 krb5kdc[27616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.65@KRBTEST.COM
I20260619 18:42:53.396173 28127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:53.386821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48232 (local address 127.21.143.65:42021)
0619 18:42:53.387054 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:53.387058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:53.387070 (+ 12us) server_negotiation.cc:407] Connection header received
0619 18:42:53.387110 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:53.387112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:53.387149 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:53.387267 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:53.387908 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:53.388788 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:53.389709 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:53.390087 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:53.390171 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:53.392504 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:53.392513 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:53.392527 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:53.392551 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:53.394367 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:53.394800 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:53.394806 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:53.394807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:53.394893 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:53.395141 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:53.395143 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:53.395144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:53.395480 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:53.395641 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:53.395834 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 18:42:53.396025 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
W20260619 18:42:53.399943 28127 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:53.399025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48238 (local address 127.21.143.65:42021)
0619 18:42:53.399159 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:53.399168 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:53.399183 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:42:53.399328 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:53.399334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:53.399384 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:53.399447 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:53.399827 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48238: BlockingRecv error: recv got EOF from 127.0.0.1:48238 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 18:42:53.400000 22077 init.cc:244] Error while opening credential cache '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/krb5cc)
Jun 19 18:42:53 dist-test-slave-mld5 krb5kdc[27616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:53.401083 22077 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 18:42:53 dist-test-slave-mld5 krb5kdc[27616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:53.402019 22077 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 18:42:53 dist-test-slave-mld5 krb5kdc[27616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.65@KRBTEST.COM
I20260619 18:42:53.410841 28127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:53.402877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48244 (local address 127.21.143.65:42021)
0619 18:42:53.403007 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:53.403011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:53.403104 (+ 93us) server_negotiation.cc:407] Connection header received
0619 18:42:53.403259 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:53.403268 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:53.403381 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:53.403464 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:53.403961 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:53.404498 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:53.405293 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:53.405543 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:53.405651 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:53.407910 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:53.407912 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:53.407914 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:53.407935 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:53.409348 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:53.409721 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:53.409723 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:53.409724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:53.409802 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:53.410027 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:53.410029 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:53.410030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:53.410181 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:53.410274 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:53.410521 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 18:42:53.410636 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 18:42:53.413908 28127 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:53.412852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48252 (local address 127.21.143.65:42021)
0619 18:42:53.412978 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:53.412982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:53.413081 (+ 99us) server_negotiation.cc:407] Connection header received
0619 18:42:53.413229 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:53.413232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:53.413302 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:53.413386 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:53.413779 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48252: BlockingRecv error: recv got EOF from 127.0.0.1:48252 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 18:42:53.414048 22077 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894542572984-22077-0/krb5kdc/krb5cc' of type FILE
Jun 19 18:42:53 dist-test-slave-mld5 krb5kdc[27616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:53.415217 22077 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 18:42:53 dist-test-slave-mld5 krb5kdc[27616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:53.416188 22077 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 18:42:53 dist-test-slave-mld5 krb5kdc[27616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.65@KRBTEST.COM
I20260619 18:42:53.425308 28127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:53.416997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48260 (local address 127.21.143.65:42021)
0619 18:42:53.417120 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:53.417124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:53.417235 (+ 111us) server_negotiation.cc:407] Connection header received
0619 18:42:53.417413 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:53.417418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:53.417460 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:53.417544 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:53.418014 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:53.418525 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:53.419353 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:53.419699 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:53.419803 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:53.422570 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:53.422574 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:53.422576 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:53.422601 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:53.423935 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:53.424332 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:53.424350 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:53.424352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:53.424418 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:53.424647 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:53.424649 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:53.424650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:53.424790 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:53.424884 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:53.425039 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 18:42:53.425118 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 18:42:53.426318 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 27723
I20260619 18:42:53.432195 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 27857
I20260619 18:42:53.437552 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 27991
I20260619 18:42:53.443166 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 27629
2026-06-19T18:42:53Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2181 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:42:53 dist-test-slave-mld5 krb5kdc[28140](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:42:53 dist-test-slave-mld5 krb5kdc[28140](info): set up 2 sockets
Jun 19 18:42:53 dist-test-slave-mld5 krb5kdc[28140](info): commencing operation
krb5kdc: starting...
W20260619 18:42:54.479975 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:42:54 dist-test-slave-mld5 krb5kdc[28140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:42:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:42:54Z Disabled control of system clock
WARNING: no policy specified for oryx/127.21.143.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.143.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.143.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.126.keytab.
Entry for principal oryx/127.21.143.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.126.keytab.
I20260619 18:42:54.600566 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:39877
--webserver_interface=127.21.143.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.126.keytab
--principal=oryx/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:40205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:39877
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:54.706643 28153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:54.706894 28153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:54.706933 28153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:54.711395 28153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:42:54.711448 28153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:54.711467 28153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:54.711488 28153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:42:54.711503 28153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:42:54.716540 28153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:40205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:39877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.126.keytab
--principal=oryx/127.21.143.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:39877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:54.717531 28153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:54.718465 28153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:54.726645 28158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:54.726619 28161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:54.726632 28159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:54.727039 28153 server_base.cc:1034] running on GCE node
I20260619 18:42:54.727701 28153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:54.728816 28153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:54.730001 28153 hybrid_clock.cc:648] HybridClock initialized: now 1781894574729987 us; error 25 us; skew 500 ppm
Jun 19 18:42:54 dist-test-slave-mld5 krb5kdc[28140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894574, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:54.732923 28153 init.cc:377] Logged in from keytab as oryx/127.21.143.126@KRBTEST.COM (short username oryx)
I20260619 18:42:54.733805 28153 webserver.cc:460] Webserver started at http://127.21.143.126:34865/ using document root <none> and password file <none>
I20260619 18:42:54.734341 28153 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:54.734400 28153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:54.734567 28153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:54.736687 28153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "81d2c79c01de40299473b15145e1bac9"
format_stamp: "Formatted at 2026-06-19 18:42:54 on dist-test-slave-mld5"
server_key: "8a9e4f053b37f370d15867a98313980d"
server_key_iv: "78aa653b18f59f190c9c233f9b635070"
server_key_version: "encryptionkey@0"
I20260619 18:42:54.737210 28153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "81d2c79c01de40299473b15145e1bac9"
format_stamp: "Formatted at 2026-06-19 18:42:54 on dist-test-slave-mld5"
server_key: "8a9e4f053b37f370d15867a98313980d"
server_key_iv: "78aa653b18f59f190c9c233f9b635070"
server_key_version: "encryptionkey@0"
I20260619 18:42:54.740933 28153 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:42:54.743266 28168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:54.744570 28153 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:42:54.744714 28153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "81d2c79c01de40299473b15145e1bac9"
format_stamp: "Formatted at 2026-06-19 18:42:54 on dist-test-slave-mld5"
server_key: "8a9e4f053b37f370d15867a98313980d"
server_key_iv: "78aa653b18f59f190c9c233f9b635070"
server_key_version: "encryptionkey@0"
I20260619 18:42:54.744823 28153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:54.766117 28153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:54.769656 28153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:54.769873 28153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:54.776966 28153 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:39877
I20260619 18:42:54.777287 28220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:39877 every 8 connection(s)
I20260619 18:42:54.778096 28153 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:42:54.780983 28221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:54.786513 28221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9: Bootstrap starting.
I20260619 18:42:54.787083 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 28153
I20260619 18:42:54.787222 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:42:54.787564 22077 external_mini_cluster.cc:1428] Setting key a0b4652f111dd95afb724d83a939b227
I20260619 18:42:54.789567 28221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:54.790310 28221 log.cc:824] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:54.792654 28221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9: No bootstrap required, opened a new log
Jun 19 18:42:54 dist-test-slave-mld5 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM
I20260619 18:42:54.796780 28221 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } }
I20260619 18:42:54.797015 28221 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:54.797073 28221 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81d2c79c01de40299473b15145e1bac9, State: Initialized, Role: FOLLOWER
I20260619 18:42:54.797595 28221 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } }
I20260619 18:42:54.797822 28221 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:42:54.797899 28221 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:42:54.797984 28221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:54.799235 28221 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } }
I20260619 18:42:54.799641 28221 leader_election.cc:302] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81d2c79c01de40299473b15145e1bac9; no voters:
I20260619 18:42:54.800001 28221 leader_election.cc:288] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:42:54.800114 28226 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:54.800355 28226 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [term 1 LEADER]: Becoming Leader. State: Replica: 81d2c79c01de40299473b15145e1bac9, State: Running, Role: LEADER
I20260619 18:42:54.800688 28226 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } }
I20260619 18:42:54.801398 28221 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:42:54.802649 28227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81d2c79c01de40299473b15145e1bac9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } }
I20260619 18:42:54.802649 28228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81d2c79c01de40299473b15145e1bac9. Latest consensus state: current_term: 1 leader_uuid: "81d2c79c01de40299473b15145e1bac9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } }
I20260619 18:42:54.802803 28228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:54.802803 28227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:54.803045 28224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:54.789028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33156 (local address 127.21.143.126:39877)
0619 18:42:54.789482 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:54.789502 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:54.789539 (+ 37us) server_negotiation.cc:407] Connection header received
0619 18:42:54.790279 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:54.790298 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:54.790743 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:54.791144 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:54.792061 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:54.793167 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:54.794299 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:54.794761 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:54.794874 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:54.797507 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:54.797513 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:54.797529 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:54.797557 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:54.799582 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:54.800083 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:54.800089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:54.800095 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:54.800295 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:54.800630 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:54.800633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:54.800635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:54.801119 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:54.801313 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:54.801642 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 18:42:54.801949 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":150,"threads_started":1}
I20260619 18:42:54.803328 28235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:42:54.807116 28235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:42:54.812928 28235 catalog_manager.cc:1347] Generated new cluster ID: 9bbe0baa520143faa08c929e05e9b145
I20260619 18:42:54.813014 28235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:42:54.836055 28235 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:42:54.837220 28235 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:42:54.843211 28235 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9: Generated new TSK 0
I20260619 18:42:54.843992 28235 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.21.143.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.143.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.143.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.65.keytab.
Entry for principal oryx/127.21.143.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.65.keytab.
I20260619 18:42:54.869920 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.65:0
--local_ip_for_outbound_sockets=127.21.143.65
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:39877
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.65.keytab
--principal=oryx/127.21.143.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:40205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:54.976094 28247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:54.976372 28247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:54.976444 28247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:54.980525 28247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:54.980608 28247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:54.980696 28247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.65
I20260619 18:42:54.985816 28247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:40205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.65.keytab
--principal=oryx/127.21.143.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:39877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.65
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:54.986864 28247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:54.987984 28247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:54.997717 28255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:54.997711 28253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:54.997717 28252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:54.997846 28247 server_base.cc:1034] running on GCE node
I20260619 18:42:54.998509 28247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:54.999128 28247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:55.000289 28247 hybrid_clock.cc:648] HybridClock initialized: now 1781894575000274 us; error 32 us; skew 500 ppm
Jun 19 18:42:54 dist-test-slave-mld5 krb5kdc[28140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894574, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:55.003207 28247 init.cc:377] Logged in from keytab as oryx/127.21.143.65@KRBTEST.COM (short username oryx)
I20260619 18:42:55.004164 28247 webserver.cc:460] Webserver started at http://127.21.143.65:45539/ using document root <none> and password file <none>
I20260619 18:42:55.004691 28247 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:55.004748 28247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:55.004911 28247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:55.006938 28247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-0/data/instance:
uuid: "053fb01d91d342f1af627e19f3e04f53"
format_stamp: "Formatted at 2026-06-19 18:42:55 on dist-test-slave-mld5"
server_key: "ed544b2a5deaaabf4b55f20dd7d97124"
server_key_iv: "2a0a877224dc245af0690eeaa8409270"
server_key_version: "encryptionkey@0"
I20260619 18:42:55.007557 28247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance:
uuid: "053fb01d91d342f1af627e19f3e04f53"
format_stamp: "Formatted at 2026-06-19 18:42:55 on dist-test-slave-mld5"
server_key: "ed544b2a5deaaabf4b55f20dd7d97124"
server_key_iv: "2a0a877224dc245af0690eeaa8409270"
server_key_version: "encryptionkey@0"
I20260619 18:42:55.011389 28247 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 18:42:55.013803 28262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:55.015066 28247 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:42:55.015210 28247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-0/wal
uuid: "053fb01d91d342f1af627e19f3e04f53"
format_stamp: "Formatted at 2026-06-19 18:42:55 on dist-test-slave-mld5"
server_key: "ed544b2a5deaaabf4b55f20dd7d97124"
server_key_iv: "2a0a877224dc245af0690eeaa8409270"
server_key_version: "encryptionkey@0"
I20260619 18:42:55.015350 28247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:55.038996 28247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:55.042361 28247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:55.042564 28247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:55.043155 28247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:55.043982 28247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:55.044051 28247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:55.044148 28247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:55.044201 28247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:55.052829 28247 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.65:44621
I20260619 18:42:55.052852 28375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.65:44621 every 8 connection(s)
I20260619 18:42:55.053645 28247 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
I20260619 18:42:55.056587 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 28247
I20260619 18:42:55.056725 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance
I20260619 18:42:55.057075 22077 external_mini_cluster.cc:1428] Setting key c77e610077c08095617fd827fdf35b0e
Jun 19 18:42:55 dist-test-slave-mld5 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894574, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.21.143.66@KRBTEST.COM; defaulting to no policy
I20260619 18:42:55.067559 28224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:55.055254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:33021 (local address 127.21.143.126:39877)
0619 18:42:55.055408 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:55.055411 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:55.056207 (+ 796us) server_negotiation.cc:407] Connection header received
0619 18:42:55.057838 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:55.057841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:55.057886 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:55.057965 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:55.059488 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.060068 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.061287 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.061535 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.061646 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:55.064034 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:55.064038 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:55.064041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:55.064066 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:55.065512 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:55.066004 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:55.066008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:55.066009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:55.066101 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:55.066439 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:55.066442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:55.066444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:55.066627 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:55.066732 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:55.067217 (+ 485us) server_negotiation.cc:299] Negotiation successful
0619 18:42:55.067367 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 18:42:55.068353 28378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:55.055663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.21.143.65:33021)
0619 18:42:55.056064 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:55.056103 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:55.057614 (+ 1511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:55.058142 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:55.058148 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:55.058619 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:55.059288 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:55.059341 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.060210 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.060213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:55.061051 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:55.061059 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.061678 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.061682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:55.061848 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:55.062411 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:55.062426 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:55.063872 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:55.065627 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:55.065633 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:55.065646 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:55.065900 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:55.066198 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:55.066202 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:55.066204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:55.066338 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:55.066730 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:55.066736 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:55.066982 (+ 246us) client_negotiation.cc:769] Sending connection context
0619 18:42:55.067164 (+ 182us) client_negotiation.cc:240] Negotiation successful
0619 18:42:55.067522 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.143.66@KRBTEST.COM" created.
I20260619 18:42:55.069660 28376 heartbeater.cc:347] Connected to a master server at 127.21.143.126:39877
I20260619 18:42:55.069924 28376 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:55.070511 28376 heartbeater.cc:511] Master 127.21.143.126:39877 requested a full tablet report, sending...
I20260619 18:42:55.072085 28185 ts_manager.cc:194] Registered new tserver with Master: 053fb01d91d342f1af627e19f3e04f53 (127.21.143.65:44621)
I20260619 18:42:55.073235 28185 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.143.65@KRBTEST.COM'} at 127.21.143.65:33021
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.143.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.66.keytab.
Entry for principal oryx/127.21.143.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.66.keytab.
I20260619 18:42:55.085166 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.66:0
--local_ip_for_outbound_sockets=127.21.143.66
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:39877
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.66.keytab
--principal=oryx/127.21.143.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:40205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:55.192912 28381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:55.193249 28381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:55.193298 28381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:55.197279 28381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:55.197345 28381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:55.197402 28381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.66
I20260619 18:42:55.202739 28381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:40205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.66.keytab
--principal=oryx/127.21.143.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:39877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.66
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:55.203825 28381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:55.204818 28381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 18:42:55.214538 28381 server_base.cc:1034] running on GCE node
W20260619 18:42:55.214601 28389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:55.214502 28386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:55.214520 28387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:55.215232 28381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:55.215916 28381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:55.217120 28381 hybrid_clock.cc:648] HybridClock initialized: now 1781894575217102 us; error 30 us; skew 500 ppm
Jun 19 18:42:55 dist-test-slave-mld5 krb5kdc[28140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894575, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:55.220044 28381 init.cc:377] Logged in from keytab as oryx/127.21.143.66@KRBTEST.COM (short username oryx)
I20260619 18:42:55.220902 28381 webserver.cc:460] Webserver started at http://127.21.143.66:43741/ using document root <none> and password file <none>
I20260619 18:42:55.221472 28381 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:55.221565 28381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:55.221796 28381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:55.223905 28381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-1/data/instance:
uuid: "580c2c8aeac04a6899c26151d90802ca"
format_stamp: "Formatted at 2026-06-19 18:42:55 on dist-test-slave-mld5"
server_key: "3ea17bb30e14f317e97785bf6fdd29f0"
server_key_iv: "3e4acf4887efa935209b01bb157c344c"
server_key_version: "encryptionkey@0"
I20260619 18:42:55.224469 28381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance:
uuid: "580c2c8aeac04a6899c26151d90802ca"
format_stamp: "Formatted at 2026-06-19 18:42:55 on dist-test-slave-mld5"
server_key: "3ea17bb30e14f317e97785bf6fdd29f0"
server_key_iv: "3e4acf4887efa935209b01bb157c344c"
server_key_version: "encryptionkey@0"
I20260619 18:42:55.228319 28381 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 18:42:55.230794 28396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:55.232082 28381 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:42:55.232223 28381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-1/wal
uuid: "580c2c8aeac04a6899c26151d90802ca"
format_stamp: "Formatted at 2026-06-19 18:42:55 on dist-test-slave-mld5"
server_key: "3ea17bb30e14f317e97785bf6fdd29f0"
server_key_iv: "3e4acf4887efa935209b01bb157c344c"
server_key_version: "encryptionkey@0"
I20260619 18:42:55.232335 28381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:55.242512 28381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:55.245412 28381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:55.245612 28381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:55.246194 28381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:55.246982 28381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:55.247059 28381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:55.247130 28381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:55.247167 28381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:55.256131 28381 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.66:38767
I20260619 18:42:55.256192 28509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.66:38767 every 8 connection(s)
I20260619 18:42:55.257081 28381 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
I20260619 18:42:55.260782 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 28381
I20260619 18:42:55.260918 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance
I20260619 18:42:55.261212 22077 external_mini_cluster.cc:1428] Setting key 148b5199243ed93dc35daf9545f703da
Jun 19 18:42:55 dist-test-slave-mld5 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894575, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.21.143.67@KRBTEST.COM; defaulting to no policy
I20260619 18:42:55.272183 28224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:55.259083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:41211 (local address 127.21.143.126:39877)
0619 18:42:55.259235 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:55.259240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:55.260174 (+ 934us) server_negotiation.cc:407] Connection header received
0619 18:42:55.261672 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:55.261675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:55.261720 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:55.261831 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:55.263244 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.264116 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.265337 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.265732 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.265854 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:55.268322 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:55.268328 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:55.268335 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:55.268366 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:55.269981 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:55.270474 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:55.270477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:55.270480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:55.270566 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:55.271000 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:55.271003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:55.271005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:55.271257 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:55.271421 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:55.271905 (+ 484us) server_negotiation.cc:299] Negotiation successful
0619 18:42:55.272028 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 18:42:55.273245 28512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:55.259576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.21.143.66:41211)
0619 18:42:55.260016 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:55.260056 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:55.261426 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:55.262030 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:55.262039 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:55.262555 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:55.263091 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:55.263105 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.264268 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.264275 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:55.265204 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:55.265214 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.265878 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.265880 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:55.266023 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:55.266636 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:55.266660 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:55.268145 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:55.270099 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:55.270103 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:55.270116 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:55.270368 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:55.270702 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:55.270706 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:55.270710 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:55.270883 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:55.271437 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:55.271442 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:55.271677 (+ 235us) client_negotiation.cc:769] Sending connection context
0619 18:42:55.271842 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 18:42:55.272215 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":116,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.143.67@KRBTEST.COM" created.
I20260619 18:42:55.274515 28510 heartbeater.cc:347] Connected to a master server at 127.21.143.126:39877
I20260619 18:42:55.274755 28510 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:55.275243 28510 heartbeater.cc:511] Master 127.21.143.126:39877 requested a full tablet report, sending...
I20260619 18:42:55.276319 28185 ts_manager.cc:194] Registered new tserver with Master: 580c2c8aeac04a6899c26151d90802ca (127.21.143.66:38767)
I20260619 18:42:55.276893 28185 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.143.66@KRBTEST.COM'} at 127.21.143.66:41211
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.143.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.67.keytab.
Entry for principal oryx/127.21.143.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.67.keytab.
I20260619 18:42:55.287716 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.67:0
--local_ip_for_outbound_sockets=127.21.143.67
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:39877
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.67.keytab
--principal=oryx/127.21.143.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:40205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:55.395483 28515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:55.395771 28515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:55.395818 28515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:55.399812 28515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:55.399870 28515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:55.399932 28515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.67
I20260619 18:42:55.405694 28515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:40205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.67.keytab
--principal=oryx/127.21.143.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:39877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.67
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:55.406677 28515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:55.407665 28515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 18:42:55.416957 28515 server_base.cc:1034] running on GCE node
W20260619 18:42:55.417008 28523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:55.417091 28521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:55.417008 28520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:55.417596 28515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:55.418220 28515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:55.419370 28515 hybrid_clock.cc:648] HybridClock initialized: now 1781894575419328 us; error 63 us; skew 500 ppm
Jun 19 18:42:55 dist-test-slave-mld5 krb5kdc[28140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894575, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:55.422303 28515 init.cc:377] Logged in from keytab as oryx/127.21.143.67@KRBTEST.COM (short username oryx)
I20260619 18:42:55.423173 28515 webserver.cc:460] Webserver started at http://127.21.143.67:43759/ using document root <none> and password file <none>
I20260619 18:42:55.423759 28515 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:55.423849 28515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:55.424041 28515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:55.425916 28515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-2/data/instance:
uuid: "0928d174307d4be2bf5124d798582bf0"
format_stamp: "Formatted at 2026-06-19 18:42:55 on dist-test-slave-mld5"
server_key: "6dab20ec7fe73a6f3a1679106ca3f451"
server_key_iv: "2a45c387317ec4216fa4077988f6c114"
server_key_version: "encryptionkey@0"
I20260619 18:42:55.426446 28515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance:
uuid: "0928d174307d4be2bf5124d798582bf0"
format_stamp: "Formatted at 2026-06-19 18:42:55 on dist-test-slave-mld5"
server_key: "6dab20ec7fe73a6f3a1679106ca3f451"
server_key_iv: "2a45c387317ec4216fa4077988f6c114"
server_key_version: "encryptionkey@0"
I20260619 18:42:55.431005 28515 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 18:42:55.433311 28530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:55.434415 28515 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:42:55.434525 28515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-2/wal
uuid: "0928d174307d4be2bf5124d798582bf0"
format_stamp: "Formatted at 2026-06-19 18:42:55 on dist-test-slave-mld5"
server_key: "6dab20ec7fe73a6f3a1679106ca3f451"
server_key_iv: "2a45c387317ec4216fa4077988f6c114"
server_key_version: "encryptionkey@0"
I20260619 18:42:55.434623 28515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:55.450145 28515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:55.452706 28515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:55.452884 28515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:55.453406 28515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:55.454144 28515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:55.454193 28515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:55.454255 28515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:55.454295 28515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:55.462867 28515 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.67:33201
I20260619 18:42:55.462893 28643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.67:33201 every 8 connection(s)
I20260619 18:42:55.463764 28515 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
Jun 19 18:42:55 dist-test-slave-mld5 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894575, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM
I20260619 18:42:55.473491 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 28515
I20260619 18:42:55.473608 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance
I20260619 18:42:55.473953 22077 external_mini_cluster.cc:1428] Setting key 47810ac655cd1045103c533a4689de7b
I20260619 18:42:55.476527 28224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:55.465458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:59591 (local address 127.21.143.126:39877)
0619 18:42:55.465601 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:55.465604 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:55.466373 (+ 769us) server_negotiation.cc:407] Connection header received
0619 18:42:55.467733 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:55.467736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:55.467781 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:55.467855 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:55.468895 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.469411 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.470443 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.470650 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.470729 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:55.473101 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:55.473104 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:55.473106 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:55.473128 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:55.474557 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:55.475043 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:55.475048 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:55.475050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:55.475131 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:55.475503 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:55.475509 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:55.475514 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:55.475687 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:55.475795 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:55.476250 (+ 455us) server_negotiation.cc:299] Negotiation successful
0619 18:42:55.476365 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 18:42:55.477228 28646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:55.465829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.21.143.67:59591)
0619 18:42:55.466231 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:55.466270 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:55.467541 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:55.467999 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:55.468005 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:55.468327 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:55.468775 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:55.468787 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.469507 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.469509 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:55.470344 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:55.470352 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.470799 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.470802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:55.470950 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:55.471576 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:55.471591 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:55.472951 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:55.474675 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:55.474682 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:55.474694 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:55.474938 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:55.475222 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:55.475227 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:55.475229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:55.475412 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:55.475791 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:55.475797 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:55.476025 (+ 228us) client_negotiation.cc:769] Sending connection context
0619 18:42:55.476193 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 18:42:55.476480 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":111,"threads_started":1}
I20260619 18:42:55.478230 28644 heartbeater.cc:347] Connected to a master server at 127.21.143.126:39877
I20260619 18:42:55.478471 28644 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:55.478911 28644 heartbeater.cc:511] Master 127.21.143.126:39877 requested a full tablet report, sending...
I20260619 18:42:55.479890 28185 ts_manager.cc:194] Registered new tserver with Master: 0928d174307d4be2bf5124d798582bf0 (127.21.143.67:33201)
I20260619 18:42:55.480417 28185 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.143.67@KRBTEST.COM'} at 127.21.143.67:59591
I20260619 18:42:55.487555 22077 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.21.143.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.143.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.143.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.125.keytab.
Entry for principal oryx/127.21.143.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.125.keytab.
I20260619 18:42:55.512048 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.125:39923
--webserver_interface=127.21.143.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.125.keytab
--principal=oryx/127.21.143.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:40205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:39877,127.21.143.125:39923
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:55.619469 28649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:55.619783 28649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:55.619851 28649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:55.623842 28649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:42:55.623912 28649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:55.623950 28649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:55.624007 28649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:42:55.624033 28649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:42:55.629035 28649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:40205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:39877,127.21.143.125:39923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.125.keytab
--principal=oryx/127.21.143.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.125:39923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.143.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
W20260619 18:42:55.629248 28649 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.21.143.126:39877,127.21.143.125:39923'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 18:42:55.630041 28649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:55.630976 28649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:55.639127 28657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:55.639118 28654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:55.639115 28655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:55.639271 28649 server_base.cc:1034] running on GCE node
I20260619 18:42:55.640020 28649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:55.640987 28649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:55.642140 28649 hybrid_clock.cc:648] HybridClock initialized: now 1781894575642122 us; error 31 us; skew 500 ppm
Jun 19 18:42:55 dist-test-slave-mld5 krb5kdc[28140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894575, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:55.644903 28649 init.cc:377] Logged in from keytab as oryx/127.21.143.125@KRBTEST.COM (short username oryx)
I20260619 18:42:55.645732 28649 webserver.cc:460] Webserver started at http://127.21.143.125:43861/ using document root <none> and password file <none>
I20260619 18:42:55.646284 28649 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:55.646342 28649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:55.646566 28649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:55.648802 28649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-1/data/instance:
uuid: "60382803ba744b659939734b7e645990"
format_stamp: "Formatted at 2026-06-19 18:42:55 on dist-test-slave-mld5"
server_key: "050a382ae5f8ff54ac2259174a72a745"
server_key_iv: "90effee8ab151630083ed8a326cacf88"
server_key_version: "encryptionkey@0"
I20260619 18:42:55.649451 28649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-1/wal/instance:
uuid: "60382803ba744b659939734b7e645990"
format_stamp: "Formatted at 2026-06-19 18:42:55 on dist-test-slave-mld5"
server_key: "050a382ae5f8ff54ac2259174a72a745"
server_key_iv: "90effee8ab151630083ed8a326cacf88"
server_key_version: "encryptionkey@0"
I20260619 18:42:55.653331 28649 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:42:55.655808 28664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:55.657055 28649 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:42:55.657184 28649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-1/wal
uuid: "60382803ba744b659939734b7e645990"
format_stamp: "Formatted at 2026-06-19 18:42:55 on dist-test-slave-mld5"
server_key: "050a382ae5f8ff54ac2259174a72a745"
server_key_iv: "90effee8ab151630083ed8a326cacf88"
server_key_version: "encryptionkey@0"
I20260619 18:42:55.657305 28649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:55.671211 28649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:55.674808 28649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:55.674973 28649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:55.682087 28649 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.125:39923
I20260619 18:42:55.682367 28716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.125:39923 every 8 connection(s)
I20260619 18:42:55.683135 28649 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-1/data/info.pb
I20260619 18:42:55.685947 28717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:55.687736 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 28649
I20260619 18:42:55.690443 28717 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:42:55 dist-test-slave-mld5 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.143.125@KRBTEST.COM
Jun 19 18:42:55 dist-test-slave-mld5 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894575, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.125@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM
I20260619 18:42:55.702816 28224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:55.692558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33162 (local address 127.21.143.126:39877)
0619 18:42:55.692709 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:55.692714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:55.693037 (+ 323us) server_negotiation.cc:407] Connection header received
0619 18:42:55.693165 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:55.693171 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:55.693227 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:55.693295 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:55.694121 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.694886 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.696134 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.696423 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.696543 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:55.699127 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:55.699130 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:55.699132 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:55.699157 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:55.700822 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:55.701325 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:55.701328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:55.701329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:55.701402 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:55.701702 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:55.701705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:55.701706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:55.702037 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:55.702144 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:55.702487 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 18:42:55.702617 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 18:42:55.703645 28720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:55.689319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47156 (local address 127.21.143.125:39923)
0619 18:42:55.689763 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:55.689785 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:55.689829 (+ 44us) server_negotiation.cc:407] Connection header received
0619 18:42:55.690692 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:55.690718 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:55.691168 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:55.691558 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:55.692247 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.693723 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.694908 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.695677 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.695835 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:55.698440 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:55.698446 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:55.698457 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:55.698507 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:55.700100 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:55.700606 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:55.700612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:55.700618 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:55.700722 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:55.701015 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:55.701021 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:55.701022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:55.701585 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:55.701811 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:55.702171 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 18:42:55.702485 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":117,"threads_started":1}
I20260619 18:42:55.703645 28721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:55.692489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.0.0.1:33162)
0619 18:42:55.692902 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:55.692952 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:55.693070 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:55.693366 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:55.693372 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:55.693840 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:55.693968 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:55.693974 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.695011 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.695014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:55.696002 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:55.696009 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.696574 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.696578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:55.696659 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:55.697186 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:55.697202 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:55.698969 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:55.700951 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:55.700958 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:55.700965 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:55.701223 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:55.701505 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:55.701508 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:55.701510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:55.701588 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:55.702140 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:55.702147 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:55.702254 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 18:42:55.702396 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 18:42:55.702547 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":144,"threads_started":1}
I20260619 18:42:55.705154 28717 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:42:55 dist-test-slave-mld5 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894575, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.125@KRBTEST.COM for oryx/127.21.143.125@KRBTEST.COM
I20260619 18:42:55.712759 28721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:55.705673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.125:39923 (local address 127.0.0.1:47160)
0619 18:42:55.705783 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:55.705797 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:55.705868 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:55.706069 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:55.706072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:55.706226 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:55.706304 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:55.706308 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.707338 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.707341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:55.707865 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:55.707872 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.708293 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.708297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:55.708383 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:55.708836 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:55.708849 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:55.709969 (+ 1120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:55.711436 (+ 1467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:55.711441 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:55.711443 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:55.711622 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:55.711876 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:55.711879 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:55.711880 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:55.711954 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:55.712288 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:55.712291 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:55.712366 (+ 75us) client_negotiation.cc:769] Sending connection context
0619 18:42:55.712471 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 18:42:55.712587 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 18:42:55.712759 28720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:55.705797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47160 (local address 127.21.143.125:39923)
0619 18:42:55.705876 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:55.705880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:55.705894 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:42:55.705930 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:55.705936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:55.705976 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:55.706054 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:55.706405 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.707223 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.707973 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.708196 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.708298 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:55.710111 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:55.710115 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:55.710117 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:55.710143 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:55.711284 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:55.711711 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:55.711716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:55.711718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:55.711789 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:55.712038 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:55.712043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:55.712045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:55.712196 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:55.712288 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:55.712524 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 18:42:55.712647 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 18:42:55.715945 28717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990: Bootstrap starting.
I20260619 18:42:55.718303 28717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:55.719034 28717 log.cc:824] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:55.721024 28717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990: No bootstrap required, opened a new log
I20260619 18:42:55.724134 28717 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } }
I20260619 18:42:55.724330 28717 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:55.724416 28717 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60382803ba744b659939734b7e645990, State: Initialized, Role: FOLLOWER
I20260619 18:42:55.724822 28717 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } }
I20260619 18:42:55.725291 28723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } } }
I20260619 18:42:55.725423 28723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:42:55.725986 28717 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 18:42:55.730733 28734 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:42:55.730822 28734 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:42:55.732645 28649 master_runner.cc:428] Detected that this master 60382803ba744b659939734b7e645990 is joining an existing cluster
I20260619 18:42:55.732717 28649 master_runner.cc:432] Initiating AddMaster RPC to add 127.21.143.125:39923
I20260619 18:42:55.743896 28224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:55.734998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33176 (local address 127.21.143.126:39877)
0619 18:42:55.735140 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:55.735145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:55.735512 (+ 367us) server_negotiation.cc:407] Connection header received
0619 18:42:55.735696 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:55.735700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:55.735750 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:55.735825 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:55.736403 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.736994 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.738313 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.738712 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.738812 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:55.740252 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:55.740256 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:55.740259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:55.740288 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:55.741971 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:55.742521 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:55.742525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:55.742527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:55.742615 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:55.742928 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:55.742934 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:55.742939 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:55.743142 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:55.743255 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:55.743615 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 18:42:55.743734 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 18:42:55.743913 28740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:55.735104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.0.0.1:33176)
0619 18:42:55.735427 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:55.735442 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:55.735577 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:55.735887 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:55.735890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:55.736091 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:55.736250 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:55.736255 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.737101 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.737106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:55.738191 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:55.738201 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.738840 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.738843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:55.738907 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:55.739702 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:55.739727 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:55.740118 (+ 391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:55.742094 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:55.742098 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:55.742100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:55.742415 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:55.742720 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:55.742723 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:55.742725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:55.742828 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:55.743248 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:55.743251 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:55.743411 (+ 160us) client_negotiation.cc:769] Sending connection context
0619 18:42:55.743557 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 18:42:55.743688 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":59,"spinlock_wait_cycles":768,"thread_start_us":94,"threads_started":1}
I20260619 18:42:55.744570 28741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:55.734944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.125:39923 (local address 127.0.0.1:47172)
0619 18:42:55.735390 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:55.735404 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:55.735496 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:55.735755 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:55.735757 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:55.736032 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:55.736228 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:55.736238 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.737275 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.737279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:55.738193 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:55.738201 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.739414 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.739424 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:55.739497 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:55.740241 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:55.740251 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:55.741001 (+ 750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:55.742986 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:55.742989 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:55.742991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:55.743220 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:55.743531 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:55.743535 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:55.743537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:55.743624 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:55.744058 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:55.744060 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:55.744125 (+ 65us) client_negotiation.cc:769] Sending connection context
0619 18:42:55.744238 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 18:42:55.744370 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":70,"thread_start_us":93,"threads_started":1}
I20260619 18:42:55.744597 28720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:55.735345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47172 (local address 127.21.143.125:39923)
0619 18:42:55.735427 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:55.735433 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:55.735473 (+ 40us) server_negotiation.cc:407] Connection header received
0619 18:42:55.735597 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:55.735603 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:55.735652 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:55.735749 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:55.736356 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.737162 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.738933 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.739258 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.739408 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:55.741329 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:55.741332 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:55.741334 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:55.741353 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:55.742899 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:55.743353 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:55.743357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:55.743359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:55.743431 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:55.743730 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:55.743733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:55.743735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:55.743940 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:55.744004 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:55.744301 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 18:42:55.744435 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":38}
Jun 19 18:42:55 dist-test-slave-mld5 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894574, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.126@KRBTEST.COM for oryx/127.21.143.125@KRBTEST.COM
I20260619 18:42:55.756898 28742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:55.748327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.125:39923 (local address 127.0.0.1:47182)
0619 18:42:55.748643 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:55.748691 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:55.748814 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:55.749086 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:55.749090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:55.749403 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:55.749497 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:55.749503 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.750423 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.750425 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:55.751014 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:55.751022 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.751568 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.751570 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:55.751653 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:55.752093 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:55.752103 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:55.753554 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:55.755223 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:55.755231 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:55.755235 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:55.755501 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:55.755776 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:55.755778 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:55.755779 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:55.755851 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:55.756292 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:55.756298 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:55.756414 (+ 116us) client_negotiation.cc:769] Sending connection context
0619 18:42:55.756551 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 18:42:55.756734 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":90,"threads_started":1}
I20260619 18:42:55.757315 28720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:55.748348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47182 (local address 127.21.143.125:39923)
0619 18:42:55.748448 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:55.748452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:55.748769 (+ 317us) server_negotiation.cc:407] Connection header received
0619 18:42:55.748936 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:55.748939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:55.748986 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:55.749065 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:55.749599 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.750321 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.751122 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.751469 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.751567 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:55.753686 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:55.753690 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:55.753693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:55.753716 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:55.755106 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:55.755592 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:55.755595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:55.755597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:55.755685 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:55.755965 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:55.755968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:55.755969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:55.756148 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:55.756236 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:55.757085 (+ 849us) server_negotiation.cc:299] Negotiation successful
0619 18:42:55.757183 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 18:42:55.759702 28185 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.21.143.125:39923: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "60382803ba744b659939734b7e645990"
member_type: NON_VOTER
last_known_addr {
host: "127.21.143.125"
port: 39923
}
attrs {
promote: true
}
}
dest_uuid: "81d2c79c01de40299473b15145e1bac9"
cas_config_opid_index: -1
I20260619 18:42:55.760308 28185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: NON_VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: true } }
I20260619 18:42:55.762082 28745 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 60382803ba744b659939734b7e645990 (127.21.143.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: NON_VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: true } } }
I20260619 18:42:55.762537 28691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:55.763999 28691 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [term 1 FOLLOWER]: Refusing update from remote peer 81d2c79c01de40299473b15145e1bac9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 18:42:55.764092 28745 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.21.143.125:39923
I20260619 18:42:55.764518 28649 master.cc:537] Master@127.21.143.125:39923 shutting down...
I20260619 18:42:55.765833 28744 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60382803ba744b659939734b7e645990" member_type: NON_VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 18:42:55.768458 28170 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
WARNING: no policy specified for oryx/127.21.143.124@KRBTEST.COM; defaulting to no policy
I20260619 18:42:55.768604 28746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "81d2c79c01de40299473b15145e1bac9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: NON_VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: true } } }
I20260619 18:42:55.768990 28746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [sys.catalog]: This master's current role is: LEADER
W20260619 18:42:55.769596 28170 consensus_peers.cc:487] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 -> Peer 60382803ba744b659939734b7e645990 (127.21.143.125:39923): Couldn't send request to peer 60382803ba744b659939734b7e645990. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 18:42:55.769744 28649 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:42:55.770035 28649 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:42:55.770148 28649 tablet_replica.cc:330] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.143.124@KRBTEST.COM" created.
I20260619 18:42:55.785651 28649 master.cc:552] Master@127.21.143.125:39923 shutdown complete.
I20260619 18:42:55.785723 28649 master_runner.cc:305] Clearing existing system tablet
I20260619 18:42:55.785965 28649 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:55.786039 28649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.143.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.124.keytab.
Entry for principal oryx/127.21.143.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.124.keytab.
I20260619 18:42:55.787436 28649 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 18:42:55.788295 28752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:55.788535 28649 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:55.788833 28649 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 18:42:55.788937 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.124:37717
--webserver_interface=127.21.143.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.124.keytab
--principal=oryx/127.21.143.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:40205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:39877,127.21.143.125:39923,127.21.143.124:37717
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:42:55.788924 28649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-1/wal
uuid: "60382803ba744b659939734b7e645990"
format_stamp: "Formatted at 2026-06-19 18:42:55 on dist-test-slave-mld5"
server_key: "050a382ae5f8ff54ac2259174a72a745"
server_key_iv: "90effee8ab151630083ed8a326cacf88"
server_key_version: "encryptionkey@0"
I20260619 18:42:55.791625 28649 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:42:55.793491 28649 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 18:42:55.793599 28649 log.cc:1196] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 18:42:55.793875 28649 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990: Deleting consensus metadata
I20260619 18:42:55.794059 28649 master_runner.cc:315] Copying system tablet from 127.21.143.126:39877
I20260619 18:42:55.795279 28649 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990: tablet copy: Beginning tablet copy session from remote peer at address 127.21.143.126:39877
I20260619 18:42:55.807390 28759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:55.795896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.0.0.1:33192)
0619 18:42:55.799489 (+ 3593us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:55.799504 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:55.799651 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:55.800272 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:55.800275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:55.800455 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:55.800555 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:55.800560 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.801471 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.801476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:55.802322 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:55.802332 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.802794 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.802798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:55.802893 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:55.803548 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:55.803568 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:55.803895 (+ 327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:55.805757 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:55.805762 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:55.805765 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:55.805971 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:55.806264 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:55.806268 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:55.806270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:55.806359 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:55.806797 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:55.806800 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:55.806900 (+ 100us) client_negotiation.cc:769] Sending connection context
0619 18:42:55.807008 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 18:42:55.807156 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3514,"thread_start_us":89,"threads_started":1}
I20260619 18:42:55.807432 28224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:55.795956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33192 (local address 127.21.143.126:39877)
0619 18:42:55.796089 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:55.796095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:55.799604 (+ 3509us) server_negotiation.cc:407] Connection header received
0619 18:42:55.799784 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:55.799788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:55.799843 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:55.799931 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:55.800684 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.801360 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.802439 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:55.802673 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:55.802802 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:55.804037 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:55.804041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:55.804044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:55.804068 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:55.805641 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:55.806073 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:55.806078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:55.806080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:55.806165 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:55.806501 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:55.806505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:55.806506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:55.806693 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:55.806797 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:55.807059 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 18:42:55.807167 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 18:42:55.808315 28205 tablet_copy_service.cc:140] P 81d2c79c01de40299473b15145e1bac9: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 60382803ba744b659939734b7e645990 ({username='oryx', principal='oryx/127.21.143.125@KRBTEST.COM'} at 127.0.0.1:33192)
I20260619 18:42:55.808516 28205 tablet_copy_service.cc:161] P 81d2c79c01de40299473b15145e1bac9: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 60382803ba744b659939734b7e645990 at {username='oryx', principal='oryx/127.21.143.125@KRBTEST.COM'} at 127.0.0.1:33192: session id = 60382803ba744b659939734b7e645990-00000000000000000000000000000000
I20260619 18:42:55.810300 28205 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 18:42:55.812110 28649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 18:42:55.814122 28649 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990: tablet copy: Starting download of 0 data blocks...
I20260619 18:42:55.814333 28649 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990: tablet copy: Starting download of 1 WAL segments...
I20260619 18:42:55.816012 28649 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 18:42:55.817677 28205 tablet_copy_service.cc:342] P 81d2c79c01de40299473b15145e1bac9: Request end of tablet copy session 60382803ba744b659939734b7e645990-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.21.143.125@KRBTEST.COM'} at 127.0.0.1:33192
I20260619 18:42:55.817803 28205 tablet_copy_service.cc:433] P 81d2c79c01de40299473b15145e1bac9: ending tablet copy session 60382803ba744b659939734b7e645990-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 60382803ba744b659939734b7e645990
I20260619 18:42:55.825109 28649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:55.826789 28760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:55.826898 28763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:55.827188 28761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:55.827679 28649 server_base.cc:1034] running on GCE node
I20260619 18:42:55.827836 28649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:55.828088 28649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:55.829222 28649 hybrid_clock.cc:648] HybridClock initialized: now 1781894575829201 us; error 37 us; skew 500 ppm
Jun 19 18:42:55 dist-test-slave-mld5 krb5kdc[28140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894575, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:55.830660 28649 init.cc:377] Logged in from keytab as oryx/127.21.143.125@KRBTEST.COM (short username oryx)
I20260619 18:42:55.831104 28649 webserver.cc:460] Webserver started at http://127.21.143.125:37157/ using document root <none> and password file <none>
I20260619 18:42:55.831288 28649 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:55.831389 28649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:55.832440 28649 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 18:42:55.833379 28770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:55.833645 28649 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 18:42:55.833731 28649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-1/wal
uuid: "60382803ba744b659939734b7e645990"
format_stamp: "Formatted at 2026-06-19 18:42:55 on dist-test-slave-mld5"
server_key: "050a382ae5f8ff54ac2259174a72a745"
server_key_iv: "90effee8ab151630083ed8a326cacf88"
server_key_version: "encryptionkey@0"
I20260619 18:42:55.833834 28649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:55.855347 28649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:55.857349 28649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:55.862236 28649 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.125:39923
I20260619 18:42:55.862243 28822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.125:39923 every 8 connection(s)
I20260619 18:42:55.862888 28649 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-1/data/info.pb
I20260619 18:42:55.864379 28823 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:42:55.865183 28823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990: Bootstrap starting.
I20260619 18:42:55.873391 28823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:42:55.873744 28823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990: Bootstrap complete.
I20260619 18:42:55.874279 28823 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: NON_VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: true } }
I20260619 18:42:55.874418 28823 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 60382803ba744b659939734b7e645990, State: Initialized, Role: LEARNER
I20260619 18:42:55.874533 28823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: NON_VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: true } }
I20260619 18:42:55.874894 28826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: NON_VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: true } } }
I20260619 18:42:55.875057 28826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [sys.catalog]: This master's current role is: LEARNER
I20260619 18:42:55.875063 28823 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:42:55.878463 28837 catalog_manager.cc:1259] Loaded cluster ID: 9bbe0baa520143faa08c929e05e9b145
I20260619 18:42:55.878574 28837 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990: loading cluster ID for follower catalog manager: success
I20260619 18:42:55.880615 28837 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990: acquiring CA information for follower catalog manager: success
I20260619 18:42:55.881776 28837 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 18:42:55.946127 28753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:55.946369 28753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:55.946408 28753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:55.950533 28753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:42:55.950590 28753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:55.950611 28753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:55.950668 28753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:42:55.950696 28753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:42:55.955803 28753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:40205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:39877,127.21.143.125:39923,127.21.143.124:37717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.124.keytab
--principal=oryx/127.21.143.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.124:37717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.143.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:55.956820 28753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:55.957819 28753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:55.965991 28842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:55.966015 28845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:55.965999 28843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:55.966300 28753 server_base.cc:1034] running on GCE node
I20260619 18:42:55.966919 28753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:55.968021 28753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:55.969177 28753 hybrid_clock.cc:648] HybridClock initialized: now 1781894575969165 us; error 29 us; skew 500 ppm
Jun 19 18:42:55 dist-test-slave-mld5 krb5kdc[28140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894575, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:55.971931 28753 init.cc:377] Logged in from keytab as oryx/127.21.143.124@KRBTEST.COM (short username oryx)
I20260619 18:42:55.972920 28753 webserver.cc:460] Webserver started at http://127.21.143.124:35485/ using document root <none> and password file <none>
I20260619 18:42:55.973479 28753 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:55.973542 28753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:55.973784 28753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:55.975893 28753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-2/data/instance:
uuid: "dab52bca615747c89249a864e1553522"
format_stamp: "Formatted at 2026-06-19 18:42:55 on dist-test-slave-mld5"
server_key: "5001e5fea0d2a851ee90be7f5eb44946"
server_key_iv: "46e3f285d3f39d82cd3d285abb607739"
server_key_version: "encryptionkey@0"
I20260619 18:42:55.976519 28753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-2/wal/instance:
uuid: "dab52bca615747c89249a864e1553522"
format_stamp: "Formatted at 2026-06-19 18:42:55 on dist-test-slave-mld5"
server_key: "5001e5fea0d2a851ee90be7f5eb44946"
server_key_iv: "46e3f285d3f39d82cd3d285abb607739"
server_key_version: "encryptionkey@0"
I20260619 18:42:55.980528 28753 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260619 18:42:55.983038 28852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:55.984340 28753 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:42:55.984479 28753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-2/wal
uuid: "dab52bca615747c89249a864e1553522"
format_stamp: "Formatted at 2026-06-19 18:42:55 on dist-test-slave-mld5"
server_key: "5001e5fea0d2a851ee90be7f5eb44946"
server_key_iv: "46e3f285d3f39d82cd3d285abb607739"
server_key_version: "encryptionkey@0"
I20260619 18:42:55.984596 28753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:56.012254 28753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:56.015470 28753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:56.015673 28753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:56.022732 28753 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.124:37717
I20260619 18:42:56.023044 28904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.124:37717 every 8 connection(s)
I20260619 18:42:56.023825 28753 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-2/data/info.pb
I20260619 18:42:56.026527 28905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:56.030280 28905 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:42:56.031136 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 28753
Jun 19 18:42:56 dist-test-slave-mld5 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894575, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.124@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM
Jun 19 18:42:56 dist-test-slave-mld5 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.143.124@KRBTEST.COM
I20260619 18:42:56.044922 28224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.032462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33206 (local address 127.21.143.126:39877)
0619 18:42:56.032633 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:56.032637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:56.033296 (+ 659us) server_negotiation.cc:407] Connection header received
0619 18:42:56.034482 (+ 1186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:56.034485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:56.034569 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:56.034694 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:56.035908 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.036506 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.037631 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.037858 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.037973 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.040785 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:56.040788 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:56.040790 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:56.040814 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:56.042468 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.043176 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.043179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.043181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.043262 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.043677 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.043680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.043681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.043922 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:56.044021 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:56.044592 (+ 571us) server_negotiation.cc:299] Negotiation successful
0619 18:42:56.044725 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 18:42:56.045954 28907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.032651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.0.0.1:33206)
0619 18:42:56.033149 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:56.033194 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:56.034219 (+ 1025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:56.034778 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:56.034784 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:56.035424 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:56.035741 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.035755 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.036632 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.036636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.037499 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.037508 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.038012 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.038016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.038338 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.038884 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:56.038900 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:56.040603 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:56.042655 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:56.042665 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:56.042676 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:56.043070 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:56.043410 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:56.043413 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:56.043415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:56.043594 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:56.044053 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:56.044059 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:56.044357 (+ 298us) client_negotiation.cc:769] Sending connection context
0619 18:42:56.044534 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 18:42:56.044905 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":62,"thread_start_us":156,"threads_started":1}
I20260619 18:42:56.045954 28908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.032651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46316 (local address 127.21.143.124:37717)
0619 18:42:56.033468 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:56.033479 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:56.033514 (+ 35us) server_negotiation.cc:407] Connection header received
0619 18:42:56.034219 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:56.034244 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:56.034504 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:56.034685 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:56.035723 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.036905 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.037754 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.038176 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.038310 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.041198 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:56.041203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:56.041214 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:56.041234 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:56.043137 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.043620 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.043629 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.043633 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.043715 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.044053 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.044057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.044059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.044359 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:56.044546 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:56.044793 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 18:42:56.044953 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":709,"spinlock_wait_cycles":51072,"thread_start_us":156,"threads_started":1}
I20260619 18:42:56.047216 28905 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:42:56 dist-test-slave-mld5 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894575, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.124@KRBTEST.COM for oryx/127.21.143.125@KRBTEST.COM
I20260619 18:42:56.055038 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.047835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47198 (local address 127.21.143.125:39923)
0619 18:42:56.048138 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:56.048142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:56.048156 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:42:56.048196 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:56.048199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:56.048244 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:56.048336 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:56.048697 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.049342 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.050105 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.050337 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.050442 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.052169 (+ 1727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:56.052172 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:56.052174 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:56.052196 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:56.053596 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.054031 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.054033 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.054034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.054102 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.054357 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.054360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.054361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.054504 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:56.054573 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:56.054841 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 18:42:56.054925 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":94,"threads_started":1}
I20260619 18:42:56.055064 28907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.047700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.125:39923 (local address 127.0.0.1:47198)
0619 18:42:56.047830 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:56.047843 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:56.047911 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:56.048344 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:56.048347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:56.048506 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:56.048586 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.048590 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.049445 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.049452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.049991 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.049998 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.050440 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.050444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.050535 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.050915 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:56.050926 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:56.052041 (+ 1115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:56.053698 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:56.053703 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:56.053705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:56.053934 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:56.054182 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:56.054185 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:56.054186 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:56.054273 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:56.054593 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:56.054596 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:56.054685 (+ 89us) client_negotiation.cc:769] Sending connection context
0619 18:42:56.054799 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 18:42:56.054913 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 18:42:56.055922 28905 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.143.124" port: 37717 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:42:56 dist-test-slave-mld5 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894575, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.124@KRBTEST.COM for oryx/127.21.143.124@KRBTEST.COM
I20260619 18:42:56.062906 28907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.056365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:37717 (local address 127.0.0.1:46330)
0619 18:42:56.056479 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:56.056493 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:56.056573 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:56.056783 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:56.056786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:56.056935 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:56.057011 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.057017 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.057675 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.057678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.058149 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.058156 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.058593 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.058595 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.058685 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.059047 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:56.059057 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:56.060137 (+ 1080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:56.061764 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:56.061766 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:56.061767 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:56.061933 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:56.062159 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:56.062160 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:56.062161 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:56.062226 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:56.062563 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:56.062564 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:56.062629 (+ 65us) client_negotiation.cc:769] Sending connection context
0619 18:42:56.062703 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 18:42:56.062799 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":16}
I20260619 18:42:56.063048 28908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.056389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46330 (local address 127.21.143.124:37717)
0619 18:42:56.056475 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:56.056479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:56.056537 (+ 58us) server_negotiation.cc:407] Connection header received
0619 18:42:56.056650 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:56.056652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:56.056683 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:56.056741 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:56.057112 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.057576 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.058271 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.058497 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.058585 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.060250 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:56.060253 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:56.060255 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:56.060282 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:56.061674 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.062007 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.062010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.062011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.062073 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.062322 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.062325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.062326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.062481 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:56.062564 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:56.062776 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 18:42:56.062858 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":28}
I20260619 18:42:56.066022 28905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522: Bootstrap starting.
I20260619 18:42:56.068356 28905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:56.069036 28905 log.cc:824] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:56.070878 28905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522: No bootstrap required, opened a new log
I20260619 18:42:56.074033 28905 raft_consensus.cc:348] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } } peers { permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 37717 } }
I20260619 18:42:56.074242 28905 raft_consensus.cc:374] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:56.074282 28905 raft_consensus.cc:729] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dab52bca615747c89249a864e1553522, State: Initialized, Role: FOLLOWER
I20260619 18:42:56.074714 28905 consensus_queue.cc:260] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } } peers { permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 37717 } }
I20260619 18:42:56.075182 28911 sys_catalog.cc:455] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } } peers { permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 37717 } } }
I20260619 18:42:56.075412 28911 sys_catalog.cc:458] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:42:56.076071 28905 sys_catalog.cc:564] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:42:56.076179 28376 heartbeater.cc:503] Master 127.21.143.126:39877 was elected leader, sending a full tablet report...
W20260619 18:42:56.081192 28922 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:42:56.081281 28922 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:42:56.084010 28753 master_runner.cc:428] Detected that this master dab52bca615747c89249a864e1553522 is joining an existing cluster
I20260619 18:42:56.084089 28753 master_runner.cc:432] Initiating AddMaster RPC to add 127.21.143.124:37717
I20260619 18:42:56.094785 28929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.086112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.0.0.1:33208)
0619 18:42:56.086667 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:56.086683 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:56.086767 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:56.087498 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:56.087501 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:56.087695 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:56.087811 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.087822 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.088933 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.088936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.089933 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.089943 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.090616 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.090619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.090703 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.091198 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:56.091209 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:56.091629 (+ 420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:56.093449 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:56.093452 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:56.093453 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:56.093647 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:56.093919 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:56.093921 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:56.093922 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:56.093991 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:56.094375 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:56.094377 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:56.094450 (+ 73us) client_negotiation.cc:769] Sending connection context
0619 18:42:56.094527 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 18:42:56.094627 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":138,"threads_started":1}
I20260619 18:42:56.094877 28224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.086983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33208 (local address 127.21.143.126:39877)
0619 18:42:56.087166 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:56.087171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:56.087184 (+ 13us) server_negotiation.cc:407] Connection header received
0619 18:42:56.087219 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:56.087221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:56.087267 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:56.087379 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:56.087967 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.088738 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.090063 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.090405 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.090473 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.091761 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:56.091766 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:56.091769 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:56.091795 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:56.093350 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.093737 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.093740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.093742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.093830 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.094072 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.094076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.094077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.094286 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:56.094390 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:56.094611 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 18:42:56.094725 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260619 18:42:56.099481 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 28247
I20260619 18:42:56.106947 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.65:44621
--local_ip_for_outbound_sockets=127.21.143.65
--tserver_master_addrs=127.21.143.126:39877
--webserver_port=45539
--webserver_interface=127.21.143.65
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.65.keytab
--principal=oryx/127.21.143.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:40205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.143.126:39877,127.21.143.125:39923
--tserver_master_addrs=127.21.143.126:39877,127.21.143.125:39923,127.21.143.124:37717 with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:42:56 dist-test-slave-mld5 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894574, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.126@KRBTEST.COM for oryx/127.21.143.124@KRBTEST.COM
I20260619 18:42:56.112242 28742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.098380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:37717 (local address 127.0.0.1:46344)
0619 18:42:56.098496 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:56.098514 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:56.098595 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:56.098941 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:56.098944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:56.099128 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:56.099258 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.099265 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.104378 (+ 5113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.104382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.104939 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.104948 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.105460 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.105464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.105574 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.106109 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:56.106122 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:56.107971 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:56.110617 (+ 2646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:56.110620 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:56.110622 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:56.110870 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:56.111177 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:56.111180 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:56.111183 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:56.111265 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:56.111728 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:56.111731 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:56.111824 (+ 93us) client_negotiation.cc:769] Sending connection context
0619 18:42:56.111903 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 18:42:56.112064 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 18:42:56.112877 28931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.098435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46344 (local address 127.21.143.124:37717)
0619 18:42:56.098697 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:56.098706 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:56.098719 (+ 13us) server_negotiation.cc:407] Connection header received
0619 18:42:56.098779 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:56.098782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:56.098838 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:56.098918 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:56.099392 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.104232 (+ 4840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.105067 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.105335 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.105448 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.108523 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:56.108527 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:56.108531 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:56.108553 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:56.110485 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.110999 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.111002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.111004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.111080 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.111401 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.111404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.111406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.111620 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:56.112511 (+ 891us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:56.112585 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 18:42:56.112709 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":111,"threads_started":1}
I20260619 18:42:56.113348 28930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.086724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:37717 (local address 127.0.0.1:46336)
0619 18:42:56.087050 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:56.087062 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:56.087135 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:56.087749 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:56.087752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:56.088011 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:56.088116 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.088123 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.089232 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.089234 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.090119 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.090128 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.090779 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.090783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.090864 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.108137 (+ 17273us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:56.108153 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:56.108606 (+ 453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:56.111699 (+ 3093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:56.111706 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:56.111708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:56.112075 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:56.112356 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:56.112360 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:56.112363 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:56.112441 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:56.112860 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:56.112863 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:56.112951 (+ 88us) client_negotiation.cc:769] Sending connection context
0619 18:42:56.113052 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 18:42:56.113173 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"spinlock_wait_cycles":2816,"thread_start_us":101,"threads_started":1}
I20260619 18:42:56.113798 28185 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.21.143.124:37717: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "dab52bca615747c89249a864e1553522"
member_type: NON_VOTER
last_known_addr {
host: "127.21.143.124"
port: 37717
}
attrs {
promote: true
}
}
dest_uuid: "81d2c79c01de40299473b15145e1bac9"
cas_config_opid_index: 5
I20260619 18:42:56.113905 28908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.087278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46336 (local address 127.21.143.124:37717)
0619 18:42:56.087394 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:56.087398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:56.087409 (+ 11us) server_negotiation.cc:407] Connection header received
0619 18:42:56.087439 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:56.087441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:56.087481 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:56.087551 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:56.088230 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.089006 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.090238 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.090558 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.090640 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.108745 (+ 18105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:56.108748 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:56.108751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:56.108773 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:56.111595 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.112180 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.112183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.112185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.112257 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.112516 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.112518 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.112520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.112728 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:56.112826 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:56.113643 (+ 817us) server_negotiation.cc:299] Negotiation successful
0619 18:42:56.113745 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":61,"spinlock_wait_cycles":3968}
I20260619 18:42:56.114403 28185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: NON_VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: true } } peers { permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: NON_VOTER last_known_addr { host: "127.21.143.124" port: 37717 } attrs { promote: true } }
I20260619 18:42:56.114552 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.086302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47202 (local address 127.21.143.125:39923)
0619 18:42:56.086435 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:56.086439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:56.086750 (+ 311us) server_negotiation.cc:407] Connection header received
0619 18:42:56.086938 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:56.086941 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:56.086993 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:56.087073 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:56.087855 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.088700 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.090360 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.090689 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.090755 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.109234 (+ 18479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:56.109238 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:56.109240 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:56.109282 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:56.110961 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.112193 (+ 1232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.112197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.112198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.112570 (+ 372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.112867 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.112871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.112872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.113313 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:56.113710 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:56.114285 (+ 575us) server_negotiation.cc:299] Negotiation successful
0619 18:42:56.114411 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 18:42:56.114979 28928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.086215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.125:39923 (local address 127.0.0.1:47202)
0619 18:42:56.086669 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:56.086683 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:56.086812 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:56.087203 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:56.087206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:56.087478 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:56.087713 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.087721 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.088847 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.088849 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.090235 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.090243 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.090917 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.090921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.091009 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.107973 (+ 16964us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:56.107993 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:56.109036 (+ 1043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:56.111719 (+ 2683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:56.111722 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:56.111723 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:56.112075 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:56.112687 (+ 612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:56.112689 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:56.112691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:56.112767 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:56.113768 (+ 1001us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:56.113771 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:56.113862 (+ 91us) client_negotiation.cc:769] Sending connection context
0619 18:42:56.114221 (+ 359us) client_negotiation.cc:240] Negotiation successful
0619 18:42:56.114368 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":59,"spinlock_wait_cycles":2432,"thread_start_us":239,"threads_started":1}
I20260619 18:42:56.115693 28746 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER dab52bca615747c89249a864e1553522 (127.21.143.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: NON_VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: true } } peers { permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: NON_VOTER last_known_addr { host: "127.21.143.124" port: 37717 } attrs { promote: true } } }
I20260619 18:42:56.117177 28746 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.21.143.124:37717
I20260619 18:42:56.117354 28746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "81d2c79c01de40299473b15145e1bac9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: NON_VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: true } } peers { permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: NON_VOTER last_known_addr { host: "127.21.143.124" port: 37717 } attrs { promote: true } } }
I20260619 18:42:56.117475 28746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:56.118034 28753 master.cc:537] Master@127.21.143.124:37717 shutting down...
W20260619 18:42:56.119094 28171 consensus_peers.cc:487] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 -> Peer dab52bca615747c89249a864e1553522 (127.21.143.124:37717): Couldn't send request to peer dab52bca615747c89249a864e1553522. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 18:42:56.122690 28753 raft_consensus.cc:2229] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260619 18:42:56.122987 28753 raft_consensus.cc:2258] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260619 18:42:56.123111 28753 tablet_replica.cc:330] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522: stopping tablet replica
I20260619 18:42:56.123261 28742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.117726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.125:39923 (local address 127.0.0.1:47218)
0619 18:42:56.119029 (+ 1303us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:56.119044 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:56.119124 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:56.119512 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:56.119515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:56.119531 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:56.119623 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.119630 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.120593 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.120598 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.121779 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.121789 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.122855 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.122859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.122961 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.123004 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 18:42:56.123074 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 18:42:56.123141 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1233}
I20260619 18:42:56.123736 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.117806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47218 (local address 127.21.143.125:39923)
0619 18:42:56.117929 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:56.117933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:56.119202 (+ 1269us) server_negotiation.cc:407] Connection header received
0619 18:42:56.119243 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:56.119245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:56.119318 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:56.119399 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:56.119736 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.120496 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.121924 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.122544 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.122666 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.122796 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:56.123516 (+ 720us) server_negotiation.cc:299] Negotiation successful
0619 18:42:56.123583 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:42:56.124187 28797 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [term 1 LEARNER]: Refusing update from remote peer 81d2c79c01de40299473b15145e1bac9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 18:42:56.124636 28747 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60382803ba744b659939734b7e645990" member_type: NON_VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 18:42:56.126508 28797 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER dab52bca615747c89249a864e1553522 (127.21.143.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: NON_VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: true } } peers { permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: NON_VOTER last_known_addr { host: "127.21.143.124" port: 37717 } attrs { promote: true } } }
I20260619 18:42:56.128522 28826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81d2c79c01de40299473b15145e1bac9. Latest consensus state: current_term: 1 leader_uuid: "81d2c79c01de40299473b15145e1bac9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: NON_VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: true } } peers { permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: NON_VOTER last_known_addr { host: "127.21.143.124" port: 37717 } attrs { promote: true } } }
I20260619 18:42:56.128672 28826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [sys.catalog]: This master's current role is: LEARNER
I20260619 18:42:56.128815 28826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "81d2c79c01de40299473b15145e1bac9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: NON_VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: true } } peers { permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: NON_VOTER last_known_addr { host: "127.21.143.124" port: 37717 } attrs { promote: true } } }
I20260619 18:42:56.128932 28826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [sys.catalog]: This master's current role is: LEARNER
I20260619 18:42:56.128966 28747 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9: attempting to promote NON_VOTER 60382803ba744b659939734b7e645990 to VOTER
I20260619 18:42:56.129369 28747 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: false } } peers { permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: NON_VOTER last_known_addr { host: "127.21.143.124" port: 37717 } attrs { promote: true } }
I20260619 18:42:56.130786 28796 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [term 1 LEARNER]: Refusing update from remote peer 81d2c79c01de40299473b15145e1bac9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 18:42:56.131201 28742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.130229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:37717 (local address 127.0.0.1:46346)
0619 18:42:56.131019 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:56.131105 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:37717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293}
I20260619 18:42:56.131186 28746 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60382803ba744b659939734b7e645990" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 18:42:56.132140 28935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.131765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:37717 (local address 127.0.0.1:46348)
0619 18:42:56.132007 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:56.132060 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:37717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":91,"threads_started":1}
W20260619 18:42:56.132550 28171 consensus_peers.cc:487] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 -> Peer dab52bca615747c89249a864e1553522 (127.21.143.124:37717): Couldn't send request to peer dab52bca615747c89249a864e1553522. Status: Network error: Client connection negotiation failed: client connection to 127.21.143.124:37717: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 18:42:56.133195 28745 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 60382803ba744b659939734b7e645990 (127.21.143.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: false } } peers { permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: NON_VOTER last_known_addr { host: "127.21.143.124" port: 37717 } attrs { promote: true } } }
I20260619 18:42:56.135049 28745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "81d2c79c01de40299473b15145e1bac9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: false } } peers { permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: NON_VOTER last_known_addr { host: "127.21.143.124" port: 37717 } attrs { promote: true } } }
I20260619 18:42:56.135179 28745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:56.135449 28745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "81d2c79c01de40299473b15145e1bac9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: false } } peers { permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: NON_VOTER last_known_addr { host: "127.21.143.124" port: 37717 } attrs { promote: true } } }
I20260619 18:42:56.135592 28745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:56.136010 28796 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 60382803ba744b659939734b7e645990 (127.21.143.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: false } } peers { permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: NON_VOTER last_known_addr { host: "127.21.143.124" port: 37717 } attrs { promote: true } } }
I20260619 18:42:56.137657 28934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "81d2c79c01de40299473b15145e1bac9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: false } } peers { permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: NON_VOTER last_known_addr { host: "127.21.143.124" port: 37717 } attrs { promote: true } } }
I20260619 18:42:56.137809 28934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:42:56.141850 28753 master.cc:552] Master@127.21.143.124:37717 shutdown complete.
I20260619 18:42:56.141911 28753 master_runner.cc:305] Clearing existing system tablet
I20260619 18:42:56.142174 28753 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:56.142242 28753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:56.143639 28753 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 18:42:56.144544 28937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:56.144801 28753 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:56.144932 28753 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 18:42:56.144964 28753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-2/wal
uuid: "dab52bca615747c89249a864e1553522"
format_stamp: "Formatted at 2026-06-19 18:42:55 on dist-test-slave-mld5"
server_key: "5001e5fea0d2a851ee90be7f5eb44946"
server_key_iv: "46e3f285d3f39d82cd3d285abb607739"
server_key_version: "encryptionkey@0"
I20260619 18:42:56.147172 28753 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:42:56.149667 28753 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 18:42:56.149773 28753 log.cc:1196] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 18:42:56.150023 28753 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522: Deleting consensus metadata
I20260619 18:42:56.150175 28753 master_runner.cc:315] Copying system tablet from 127.21.143.126:39877
I20260619 18:42:56.151330 28753 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522: tablet copy: Beginning tablet copy session from remote peer at address 127.21.143.126:39877
I20260619 18:42:56.160081 28943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.151875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.0.0.1:33218)
0619 18:42:56.152254 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:56.152269 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:56.152396 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:56.152661 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:56.152664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:56.152888 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:56.153009 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.153015 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.154046 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.154049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.154870 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.154881 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.155465 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.155468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.155559 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.156110 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:56.156126 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:56.156565 (+ 439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:56.158450 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:56.158454 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:56.158456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:56.158699 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:56.159008 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:56.159011 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:56.159013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:56.159106 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:56.159540 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:56.159543 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:56.159640 (+ 97us) client_negotiation.cc:769] Sending connection context
0619 18:42:56.159748 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 18:42:56.159900 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":81,"threads_started":1}
I20260619 18:42:56.160086 28224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.151970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33218 (local address 127.21.143.126:39877)
0619 18:42:56.152108 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:56.152112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:56.152370 (+ 258us) server_negotiation.cc:407] Connection header received
0619 18:42:56.152482 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:56.152485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:56.152539 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:56.152633 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:56.153126 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.153919 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.154988 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.155354 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.155453 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.156704 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:56.156708 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:56.156711 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:56.156735 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:56.158337 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.158820 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.158824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.158826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.158910 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.159198 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.159201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.159203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.159444 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:56.159547 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:56.159830 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 18:42:56.159939 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 18:42:56.160790 28205 tablet_copy_service.cc:140] P 81d2c79c01de40299473b15145e1bac9: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer dab52bca615747c89249a864e1553522 ({username='oryx', principal='oryx/127.21.143.124@KRBTEST.COM'} at 127.0.0.1:33218)
I20260619 18:42:56.160880 28205 tablet_copy_service.cc:161] P 81d2c79c01de40299473b15145e1bac9: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer dab52bca615747c89249a864e1553522 at {username='oryx', principal='oryx/127.21.143.124@KRBTEST.COM'} at 127.0.0.1:33218: session id = dab52bca615747c89249a864e1553522-00000000000000000000000000000000
I20260619 18:42:56.161959 28205 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 18:42:56.163282 28753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 18:42:56.165508 28753 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522: tablet copy: Starting download of 0 data blocks...
I20260619 18:42:56.165678 28753 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522: tablet copy: Starting download of 1 WAL segments...
I20260619 18:42:56.167157 28753 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 18:42:56.168843 28205 tablet_copy_service.cc:342] P 81d2c79c01de40299473b15145e1bac9: Request end of tablet copy session dab52bca615747c89249a864e1553522-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.21.143.124@KRBTEST.COM'} at 127.0.0.1:33218
I20260619 18:42:56.168926 28205 tablet_copy_service.cc:433] P 81d2c79c01de40299473b15145e1bac9: ending tablet copy session dab52bca615747c89249a864e1553522-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer dab52bca615747c89249a864e1553522
I20260619 18:42:56.176148 28753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:56.177680 28944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:56.180078 28945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:56.180124 28947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:56.185989 28753 server_base.cc:1034] running on GCE node
I20260619 18:42:56.186187 28753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:56.186475 28753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:56.187649 28753 hybrid_clock.cc:648] HybridClock initialized: now 1781894576187590 us; error 41 us; skew 500 ppm
Jun 19 18:42:56 dist-test-slave-mld5 krb5kdc[28140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894576, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:56.189229 28753 init.cc:377] Logged in from keytab as oryx/127.21.143.124@KRBTEST.COM (short username oryx)
I20260619 18:42:56.189710 28753 webserver.cc:460] Webserver started at http://127.21.143.124:45219/ using document root <none> and password file <none>
I20260619 18:42:56.189931 28753 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:56.189987 28753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:56.191356 28753 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 18:42:56.192332 28954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:56.192567 28753 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 18:42:56.192642 28753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-2/wal
uuid: "dab52bca615747c89249a864e1553522"
format_stamp: "Formatted at 2026-06-19 18:42:55 on dist-test-slave-mld5"
server_key: "5001e5fea0d2a851ee90be7f5eb44946"
server_key_iv: "46e3f285d3f39d82cd3d285abb607739"
server_key_version: "encryptionkey@0"
I20260619 18:42:56.192721 28753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:56.230008 28753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:56.232180 28753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:56.237397 29006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.124:37717 every 8 connection(s)
I20260619 18:42:56.237375 28753 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.124:37717
I20260619 18:42:56.238139 28753 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/master-2/data/info.pb
I20260619 18:42:56.239962 29007 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:42:56.241048 29007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522: Bootstrap starting.
I20260619 18:42:56.253496 29007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:42:56.253795 29007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522: Bootstrap complete.
I20260619 18:42:56.254328 29007 raft_consensus.cc:348] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: false } } peers { permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: NON_VOTER last_known_addr { host: "127.21.143.124" port: 37717 } attrs { promote: true } }
I20260619 18:42:56.254465 29007 raft_consensus.cc:729] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dab52bca615747c89249a864e1553522, State: Initialized, Role: LEARNER
I20260619 18:42:56.254558 29007 consensus_queue.cc:260] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: false } } peers { permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: NON_VOTER last_known_addr { host: "127.21.143.124" port: 37717 } attrs { promote: true } }
I20260619 18:42:56.254877 29010 sys_catalog.cc:455] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: false } } peers { permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: NON_VOTER last_known_addr { host: "127.21.143.124" port: 37717 } attrs { promote: true } } }
I20260619 18:42:56.255043 29007 sys_catalog.cc:564] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:42:56.255110 29010 sys_catalog.cc:458] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [sys.catalog]: This master's current role is: LEARNER
I20260619 18:42:56.257869 29021 catalog_manager.cc:1259] Loaded cluster ID: 9bbe0baa520143faa08c929e05e9b145
I20260619 18:42:56.260308 29021 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522: loading cluster ID for follower catalog manager: success
I20260619 18:42:56.262322 29021 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522: acquiring CA information for follower catalog manager: success
I20260619 18:42:56.263473 29021 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 18:42:56.272215 28932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:56.272495 28932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:56.272603 28932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:56.276862 28932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:56.276934 28932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:56.277035 28932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.65
I20260619 18:42:56.278858 28510 heartbeater.cc:503] Master 127.21.143.126:39877 was elected leader, sending a full tablet report...
I20260619 18:42:56.282306 28932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:40205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.65.keytab
--principal=oryx/127.21.143.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.65:44621
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.143.65
--webserver_port=45539
--tserver_master_addrs=127.21.143.126:39877,127.21.143.125:39923,127.21.143.124:37717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.65
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:56.283349 28932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:56.284337 28932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:56.293843 29027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:56.293852 29026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:56.293850 29029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:56.294013 28932 server_base.cc:1034] running on GCE node
I20260619 18:42:56.294525 28932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:56.295092 28932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:56.296277 28932 hybrid_clock.cc:648] HybridClock initialized: now 1781894576296248 us; error 46 us; skew 500 ppm
Jun 19 18:42:56 dist-test-slave-mld5 krb5kdc[28140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894576, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:56.298893 28932 init.cc:377] Logged in from keytab as oryx/127.21.143.65@KRBTEST.COM (short username oryx)
I20260619 18:42:56.299779 28932 webserver.cc:460] Webserver started at http://127.21.143.65:45539/ using document root <none> and password file <none>
I20260619 18:42:56.300320 28932 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:56.300423 28932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:56.304466 28932 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260619 18:42:56.306474 29036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:56.307879 28932 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:42:56.308022 28932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-0/wal
uuid: "053fb01d91d342f1af627e19f3e04f53"
format_stamp: "Formatted at 2026-06-19 18:42:55 on dist-test-slave-mld5"
server_key: "ed544b2a5deaaabf4b55f20dd7d97124"
server_key_iv: "2a0a877224dc245af0690eeaa8409270"
server_key_version: "encryptionkey@0"
I20260619 18:42:56.308615 28932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:56.327658 28932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:56.331270 28932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:56.331594 28932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:56.332269 28932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:56.333194 28932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:56.333266 28932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:56.333334 28932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:56.333386 28932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:56.343459 28932 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.65:44621
I20260619 18:42:56.343511 29149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.65:44621 every 8 connection(s)
I20260619 18:42:56.344529 28932 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
I20260619 18:42:56.352867 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 28932
I20260619 18:42:56.353072 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 28381
Jun 19 18:42:56 dist-test-slave-mld5 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894576, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.124@KRBTEST.COM
Jun 19 18:42:56 dist-test-slave-mld5 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894576, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.125@KRBTEST.COM
Jun 19 18:42:56 dist-test-slave-mld5 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894576, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM
I20260619 18:42:56.361169 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.66:38767
--local_ip_for_outbound_sockets=127.21.143.66
--tserver_master_addrs=127.21.143.126:39877
--webserver_port=43741
--webserver_interface=127.21.143.66
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.66.keytab
--principal=oryx/127.21.143.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:40205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.143.126:39877,127.21.143.125:39923
--tserver_master_addrs=127.21.143.126:39877,127.21.143.125:39923,127.21.143.124:37717 with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:42:56.362586 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.346330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:45493 (local address 127.21.143.125:39923)
0619 18:42:56.346511 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:56.346518 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:56.347398 (+ 880us) server_negotiation.cc:407] Connection header received
0619 18:42:56.348707 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:56.348710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:56.348763 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:56.348847 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:56.350355 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.351138 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.352428 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.352772 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.352864 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.357114 (+ 4250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:56.357117 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:56.357120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:56.357146 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:56.359222 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.361059 (+ 1837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.361062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.361063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.361132 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.361711 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.361714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.361716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.361917 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:56.362001 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:56.362309 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 18:42:56.362425 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 18:42:56.362586 28224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.346935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:47501 (local address 127.21.143.126:39877)
0619 18:42:56.347069 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:56.347073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:56.347553 (+ 480us) server_negotiation.cc:407] Connection header received
0619 18:42:56.348987 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:56.348991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:56.349045 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:56.349148 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:56.350342 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.350881 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.352429 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.352775 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.352885 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.357819 (+ 4934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:56.357843 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:56.357849 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:56.357879 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:56.360043 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.360550 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.360554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.360556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.360632 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.361046 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.361049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.361051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.361321 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:56.361427 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:56.362309 (+ 882us) server_negotiation.cc:299] Negotiation successful
0619 18:42:56.362425 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:42:56.363586 29154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.346737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.125:39923 (local address 127.21.143.65:45493)
0619 18:42:56.347232 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:56.347273 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:56.348492 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:56.349053 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:56.349062 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:56.349631 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:56.350212 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.350220 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.351243 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.351249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.352265 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.352292 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.353054 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.353057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.354295 (+ 1238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.354837 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:56.354852 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:56.356940 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:56.359394 (+ 2454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:56.359402 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:56.359417 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:56.360901 (+ 1484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:56.361241 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:56.361494 (+ 253us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:56.361496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:56.361582 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:56.362054 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:56.362056 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:56.362131 (+ 75us) client_negotiation.cc:769] Sending connection context
0619 18:42:56.362988 (+ 857us) client_negotiation.cc:240] Negotiation successful
0619 18:42:56.363128 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":123,"spinlock_wait_cycles":42240,"thread_start_us":129,"threads_started":1}
I20260619 18:42:56.363895 29155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.346737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.21.143.65:47501)
0619 18:42:56.347475 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:56.347487 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:56.348509 (+ 1022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:56.349189 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:56.349192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:56.349643 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:56.350199 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.350213 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.351013 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.351016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.352265 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.352292 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.353028 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.353033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.355443 (+ 2410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.355923 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:56.355940 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:56.357656 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:56.360185 (+ 2529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:56.360189 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:56.360191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:56.360444 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:56.360721 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:56.360723 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:56.360725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:56.360895 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:56.361481 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:56.361488 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:56.361819 (+ 331us) client_negotiation.cc:769] Sending connection context
0619 18:42:56.362014 (+ 195us) client_negotiation.cc:240] Negotiation successful
0619 18:42:56.362394 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"mutex_wait_us":183,"spinlock_wait_cycles":46720,"thread_start_us":136,"threads_started":1}
I20260619 18:42:56.364549 29157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.347987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:60391 (local address 127.21.143.124:37717)
0619 18:42:56.348244 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:56.348253 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:56.348266 (+ 13us) server_negotiation.cc:407] Connection header received
0619 18:42:56.348815 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:56.348821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:56.348875 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:56.348959 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:56.350355 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.351428 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.352346 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.352679 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.352773 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.358583 (+ 5810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:56.358589 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:56.358595 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:56.358640 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:56.360389 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.360883 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.360887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.360889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.360976 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.361257 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.361260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.361262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.361532 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:56.361626 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:56.364254 (+ 2628us) server_negotiation.cc:299] Negotiation successful
0619 18:42:56.364376 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":71,"threads_started":1}
I20260619 18:42:56.364933 29151 heartbeater.cc:347] Connected to a master server at 127.21.143.126:39877
I20260619 18:42:56.364969 29152 heartbeater.cc:347] Connected to a master server at 127.21.143.125:39923
I20260619 18:42:56.365001 29156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.347845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:37717 (local address 127.21.143.65:60391)
0619 18:42:56.348135 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:56.348148 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:56.348522 (+ 374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:56.349053 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:56.349062 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:56.349631 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:56.350214 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.350220 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.351524 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.351529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.352208 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.352216 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.352947 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.352951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.353659 (+ 708us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.354313 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:56.354343 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:56.358426 (+ 4083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:56.360508 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:56.360511 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:56.360513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:56.360768 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:56.361067 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:56.361073 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:56.361077 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:56.361156 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:56.361685 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:56.361691 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:56.361828 (+ 137us) client_negotiation.cc:769] Sending connection context
0619 18:42:56.362012 (+ 184us) client_negotiation.cc:240] Negotiation successful
0619 18:42:56.364838 (+ 2826us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":129,"spinlock_wait_cycles":19072,"thread_start_us":95,"threads_started":1}
I20260619 18:42:56.365185 29152 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:56.365185 29151 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:56.365582 29150 heartbeater.cc:347] Connected to a master server at 127.21.143.124:37717
I20260619 18:42:56.365669 29150 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:56.365809 29151 heartbeater.cc:511] Master 127.21.143.126:39877 requested a full tablet report, sending...
I20260619 18:42:56.365813 29152 heartbeater.cc:511] Master 127.21.143.125:39923 requested a full tablet report, sending...
I20260619 18:42:56.367017 28185 ts_manager.cc:194] Re-registered known tserver with Master: 053fb01d91d342f1af627e19f3e04f53 (127.21.143.65:44621)
I20260619 18:42:56.365814 29150 heartbeater.cc:511] Master 127.21.143.124:37717 requested a full tablet report, sending...
I20260619 18:42:56.367488 28787 ts_manager.cc:194] Registered new tserver with Master: 053fb01d91d342f1af627e19f3e04f53 (127.21.143.65:44621)
I20260619 18:42:56.367812 28185 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.143.65@KRBTEST.COM'} at 127.21.143.65:47501
I20260619 18:42:56.368417 28971 ts_manager.cc:194] Registered new tserver with Master: 053fb01d91d342f1af627e19f3e04f53 (127.21.143.65:44621)
W20260619 18:42:56.478615 29158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:56.478892 29158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:56.478937 29158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 18:42:56.482172 28644 heartbeater.cc:503] Master 127.21.143.126:39877 was elected leader, sending a full tablet report...
W20260619 18:42:56.483114 29158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:56.483191 29158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:56.483340 29158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.66
I20260619 18:42:56.488704 29158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:40205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.66.keytab
--principal=oryx/127.21.143.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.66:38767
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.143.66
--webserver_port=43741
--tserver_master_addrs=127.21.143.126:39877,127.21.143.125:39923,127.21.143.124:37717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.66
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:56.489774 29158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:56.490793 29158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:56.500488 29166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:56.500562 29163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:56.500509 29164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:56.500674 29158 server_base.cc:1034] running on GCE node
I20260619 18:42:56.501134 29158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:56.501859 29158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:56.503042 29158 hybrid_clock.cc:648] HybridClock initialized: now 1781894576503029 us; error 35 us; skew 500 ppm
Jun 19 18:42:56 dist-test-slave-mld5 krb5kdc[28140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894576, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:56.505901 29158 init.cc:377] Logged in from keytab as oryx/127.21.143.66@KRBTEST.COM (short username oryx)
I20260619 18:42:56.506809 29158 webserver.cc:460] Webserver started at http://127.21.143.66:43741/ using document root <none> and password file <none>
I20260619 18:42:56.507431 29158 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:56.507532 29158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:56.511233 29158 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:42:56.513201 29173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:56.514389 29158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:42:56.514546 29158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-1/wal
uuid: "580c2c8aeac04a6899c26151d90802ca"
format_stamp: "Formatted at 2026-06-19 18:42:55 on dist-test-slave-mld5"
server_key: "3ea17bb30e14f317e97785bf6fdd29f0"
server_key_iv: "3e4acf4887efa935209b01bb157c344c"
server_key_version: "encryptionkey@0"
I20260619 18:42:56.515075 29158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:56.533046 29158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:56.536824 29158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:56.537045 29158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:56.537743 29158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:56.538973 29158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:56.539047 29158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:56.539124 29158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:56.539170 29158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:56.548497 29158 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.66:38767
I20260619 18:42:56.548535 29286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.66:38767 every 8 connection(s)
I20260619 18:42:56.549523 29158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
I20260619 18:42:56.559608 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 29158
I20260619 18:42:56.559796 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 28515
Jun 19 18:42:56 dist-test-slave-mld5 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894576, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.124@KRBTEST.COM
Jun 19 18:42:56 dist-test-slave-mld5 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894576, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.125@KRBTEST.COM
Jun 19 18:42:56 dist-test-slave-mld5 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894576, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM
I20260619 18:42:56.569173 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.553462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:52183 (local address 127.21.143.125:39923)
0619 18:42:56.553586 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:56.553591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:56.553919 (+ 328us) server_negotiation.cc:407] Connection header received
0619 18:42:56.554064 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:56.554067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:56.554116 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:56.554178 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:56.555870 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.556722 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.558138 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.558453 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.558557 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.561878 (+ 3321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:56.561884 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:56.561888 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:56.561917 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:56.564189 (+ 2272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.566608 (+ 2419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.566614 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.566619 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.566701 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.567476 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.567483 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.567487 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.567755 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:56.567830 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:56.568792 (+ 962us) server_negotiation.cc:299] Negotiation successful
0619 18:42:56.568938 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:42:56.569684 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.67:33201
--local_ip_for_outbound_sockets=127.21.143.67
--tserver_master_addrs=127.21.143.126:39877
--webserver_port=43759
--webserver_interface=127.21.143.67
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.67.keytab
--principal=oryx/127.21.143.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:40205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.143.126:39877,127.21.143.125:39923
--tserver_master_addrs=127.21.143.126:39877,127.21.143.125:39923,127.21.143.124:37717 with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:42:56.570880 28224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.551471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:36217 (local address 127.21.143.126:39877)
0619 18:42:56.551615 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:56.551620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:56.552967 (+ 1347us) server_negotiation.cc:407] Connection header received
0619 18:42:56.554036 (+ 1069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:56.554041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:56.554091 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:56.554171 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:56.555909 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.556722 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.557854 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.558106 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.558213 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.566155 (+ 7942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:56.566159 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:56.566162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:56.566212 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:56.568426 (+ 2214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.568976 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.568980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.568982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.569068 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.569661 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.569664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.569666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.570219 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:56.570305 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:56.570609 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 18:42:56.570735 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 18:42:56.570907 29293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.553304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.125:39923 (local address 127.21.143.66:52183)
0619 18:42:56.553829 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:56.553840 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:56.553954 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:56.554372 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:56.554381 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:56.555027 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:56.555746 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.555753 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.556830 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.556833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.557995 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.558004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.558722 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.558727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.558819 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.559342 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:56.559360 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:56.561692 (+ 2332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:56.565662 (+ 3970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:56.565668 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:56.565673 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:56.566208 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:56.566811 (+ 603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:56.566817 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:56.566822 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:56.566925 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:56.568491 (+ 1566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:56.568497 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:56.568601 (+ 104us) client_negotiation.cc:769] Sending connection context
0619 18:42:56.569347 (+ 746us) client_negotiation.cc:240] Negotiation successful
0619 18:42:56.569700 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":263,"spinlock_wait_cycles":18944,"thread_start_us":115,"threads_started":1}
I20260619 18:42:56.571818 29157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.553898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:37941 (local address 127.21.143.124:37717)
0619 18:42:56.554031 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:56.554035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:56.554291 (+ 256us) server_negotiation.cc:407] Connection header received
0619 18:42:56.554388 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:56.554391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:56.554438 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:56.554513 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:56.555899 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.556531 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.558116 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.558431 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.558540 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.562310 (+ 3770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:56.562317 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:56.562323 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:56.562356 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:56.565045 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.565909 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.565916 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.565921 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.566008 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.567036 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.567042 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.567047 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.567328 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:56.567409 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:56.571418 (+ 4009us) server_negotiation.cc:299] Negotiation successful
0619 18:42:56.571602 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:42:56.572422 29289 heartbeater.cc:347] Connected to a master server at 127.21.143.125:39923
I20260619 18:42:56.572715 29289 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:56.572907 29291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.551975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.21.143.66:36217)
0619 18:42:56.552811 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:56.552856 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:56.553818 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:56.554371 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:56.554382 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:56.555016 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:56.555716 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.555748 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.556870 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.556875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.557720 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.557729 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.558241 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.558244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.558417 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.563830 (+ 5413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:56.563844 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:56.565944 (+ 2100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:56.568585 (+ 2641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:56.568588 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:56.568590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:56.568863 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:56.569210 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:56.569213 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:56.569215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:56.569540 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:56.570360 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:56.570363 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:56.570443 (+ 80us) client_negotiation.cc:769] Sending connection context
0619 18:42:56.571733 (+ 1290us) client_negotiation.cc:240] Negotiation successful
0619 18:42:56.572106 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"mutex_wait_us":85,"thread_start_us":528,"threads_started":1}
I20260619 18:42:56.573341 29289 heartbeater.cc:511] Master 127.21.143.125:39923 requested a full tablet report, sending...
I20260619 18:42:56.573674 29288 heartbeater.cc:347] Connected to a master server at 127.21.143.126:39877
I20260619 18:42:56.573767 29288 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:56.573942 29288 heartbeater.cc:511] Master 127.21.143.126:39877 requested a full tablet report, sending...
I20260619 18:42:56.574101 29292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.553603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:37717 (local address 127.21.143.66:37941)
0619 18:42:56.554211 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:56.554224 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:56.554314 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:56.554584 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:56.554587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:56.555027 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:56.555723 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.555733 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.556673 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.556678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.557994 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.558004 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.558552 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.558555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.558646 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.559115 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:56.559130 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:56.562156 (+ 3026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:56.565186 (+ 3030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:56.565194 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:56.565212 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:56.565567 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:56.566315 (+ 748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:56.566321 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:56.566326 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:56.566498 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:56.567925 (+ 1427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:56.567934 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:56.568248 (+ 314us) client_negotiation.cc:769] Sending connection context
0619 18:42:56.572052 (+ 3804us) client_negotiation.cc:240] Negotiation successful
0619 18:42:56.572249 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":211,"spinlock_wait_cycles":4480,"thread_start_us":90,"threads_started":1}
I20260619 18:42:56.574594 28185 ts_manager.cc:194] Re-registered known tserver with Master: 580c2c8aeac04a6899c26151d90802ca (127.21.143.66:38767)
I20260619 18:42:56.574676 28787 ts_manager.cc:194] Registered new tserver with Master: 580c2c8aeac04a6899c26151d90802ca (127.21.143.66:38767)
I20260619 18:42:56.575488 28185 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.143.66@KRBTEST.COM'} at 127.21.143.66:36217
I20260619 18:42:56.575762 29287 heartbeater.cc:347] Connected to a master server at 127.21.143.124:37717
I20260619 18:42:56.575847 29287 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:56.576030 29287 heartbeater.cc:511] Master 127.21.143.124:37717 requested a full tablet report, sending...
I20260619 18:42:56.576802 28971 ts_manager.cc:194] Registered new tserver with Master: 580c2c8aeac04a6899c26151d90802ca (127.21.143.66:38767)
I20260619 18:42:56.630226 29157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.626417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60456 (local address 127.21.143.124:37717)
0619 18:42:56.626552 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:56.626557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:56.626619 (+ 62us) server_negotiation.cc:407] Connection header received
0619 18:42:56.626773 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:56.626776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:56.626828 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:56.626917 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:56.627215 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.627792 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.629272 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.629691 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.629793 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.629895 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:56.630044 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 18:42:56.630106 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 18:42:56.630252 28742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.626328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:37717 (local address 127.0.0.1:60456)
0619 18:42:56.626533 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:56.626548 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:56.626663 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:56.626966 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:56.626969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:56.626990 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:56.627094 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.627100 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.627909 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.627912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.629121 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.629134 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.629820 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.629824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.629917 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.629943 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 18:42:56.630023 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 18:42:56.630103 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260619 18:42:56.631108 28981 raft_consensus.cc:1203] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 18:42:56.631570 29010 sys_catalog.cc:455] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81d2c79c01de40299473b15145e1bac9. Latest consensus state: current_term: 1 leader_uuid: "81d2c79c01de40299473b15145e1bac9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: false } } peers { permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: NON_VOTER last_known_addr { host: "127.21.143.124" port: 37717 } attrs { promote: true } } }
I20260619 18:42:56.631714 29010 sys_catalog.cc:458] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [sys.catalog]: This master's current role is: LEARNER
W20260619 18:42:56.689128 29294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:56.689388 29294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:56.689429 29294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:56.693521 29294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:56.693574 29294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:56.693630 29294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.67
I20260619 18:42:56.698757 29294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:40205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.67.keytab
--principal=oryx/127.21.143.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.67:33201
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.143.67
--webserver_port=43759
--tserver_master_addrs=127.21.143.126:39877,127.21.143.125:39923,127.21.143.124:37717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.67
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:56.699788 29294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:56.700748 29294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:56.710471 29302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:56.710494 29299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:56.710626 29294 server_base.cc:1034] running on GCE node
W20260619 18:42:56.710518 29300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:56.711155 29294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:56.711783 29294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:56.712965 29294 hybrid_clock.cc:648] HybridClock initialized: now 1781894576712949 us; error 27 us; skew 500 ppm
Jun 19 18:42:56 dist-test-slave-mld5 krb5kdc[28140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894576, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:56.715767 29294 init.cc:377] Logged in from keytab as oryx/127.21.143.67@KRBTEST.COM (short username oryx)
I20260619 18:42:56.716622 29294 webserver.cc:460] Webserver started at http://127.21.143.67:43759/ using document root <none> and password file <none>
I20260619 18:42:56.717185 29294 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:56.717288 29294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:56.721504 29294 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:42:56.723559 29309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:56.724721 29294 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 18:42:56.724850 29294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-2/wal
uuid: "0928d174307d4be2bf5124d798582bf0"
format_stamp: "Formatted at 2026-06-19 18:42:55 on dist-test-slave-mld5"
server_key: "6dab20ec7fe73a6f3a1679106ca3f451"
server_key_iv: "2a45c387317ec4216fa4077988f6c114"
server_key_version: "encryptionkey@0"
I20260619 18:42:56.725386 29294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:56.756309 29294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:56.759722 29294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:56.759927 29294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:56.760542 29294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:56.761421 29294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:56.761490 29294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:56.761559 29294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:56.761611 29294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:56.770705 29294 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.67:33201
I20260619 18:42:56.770727 29422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.67:33201 every 8 connection(s)
I20260619 18:42:56.771888 29294 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
I20260619 18:42:56.781261 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 29294
Jun 19 18:42:56 dist-test-slave-mld5 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894576, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.125@KRBTEST.COM
Jun 19 18:42:56 dist-test-slave-mld5 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894576, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.124@KRBTEST.COM
Jun 19 18:42:56 dist-test-slave-mld5 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894576, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM
I20260619 18:42:56.789178 29157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.775575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:38419 (local address 127.21.143.124:37717)
0619 18:42:56.775705 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:56.775711 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:56.775847 (+ 136us) server_negotiation.cc:407] Connection header received
0619 18:42:56.776398 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:56.776401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:56.776461 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:56.776534 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:56.778015 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.778780 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.780447 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.780799 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.780867 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.784907 (+ 4040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:56.784911 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:56.784914 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:56.784946 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:56.786943 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.787551 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.787555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.787557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.787635 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.788007 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.788010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.788012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.788213 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:56.788304 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:56.788904 (+ 600us) server_negotiation.cc:299] Negotiation successful
0619 18:42:56.789023 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 18:42:56.789460 28224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.775851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:41997 (local address 127.21.143.126:39877)
0619 18:42:56.775965 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:56.775968 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:56.775981 (+ 13us) server_negotiation.cc:407] Connection header received
0619 18:42:56.776400 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:56.776404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:56.776460 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:56.776541 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:56.779243 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.780058 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.782641 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.783019 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.783100 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.785893 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:56.785897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:56.785900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:56.785932 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:56.787745 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.788210 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.788215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.788216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.788297 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.788606 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.788611 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.788613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.788816 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:56.788881 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:56.789202 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 18:42:56.789309 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 18:42:56.790309 29427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.774256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.21.143.67:41997)
0619 18:42:56.774758 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:56.774845 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:56.776141 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:56.778741 (+ 2600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:56.778745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:56.778949 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:56.779098 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.779105 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.780204 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.780207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.782501 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.782512 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.783286 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.783292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.783409 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.783906 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:56.783926 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:56.785768 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:56.787865 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:56.787868 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:56.787869 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:56.788099 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:56.788425 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:56.788428 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:56.788430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:56.788506 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:56.788949 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:56.788952 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:56.789026 (+ 74us) client_negotiation.cc:769] Sending connection context
0619 18:42:56.789722 (+ 696us) client_negotiation.cc:240] Negotiation successful
0619 18:42:56.789860 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":32,"spinlock_wait_cycles":10368,"thread_start_us":127,"threads_started":1}
I20260619 18:42:56.791034 29428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.774256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.125:39923 (local address 127.21.143.67:57347)
0619 18:42:56.774758 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:56.774817 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:56.776140 (+ 1323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:56.776763 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:56.776771 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:56.777293 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:56.777864 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.777879 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.778706 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.778710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.780312 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.780320 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.781064 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.781068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.781273 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.781980 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:56.782001 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:56.785142 (+ 3141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:56.787507 (+ 2365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:56.787511 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:56.787514 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:56.787765 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:56.788076 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:56.788078 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:56.788080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:56.788164 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:56.788629 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:56.788632 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:56.788719 (+ 87us) client_negotiation.cc:769] Sending connection context
0619 18:42:56.788848 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 18:42:56.789220 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":128,"spinlock_wait_cycles":51968,"thread_start_us":143,"threads_started":1}
I20260619 18:42:56.790293 29429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.775424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:37717 (local address 127.21.143.67:38419)
0619 18:42:56.775747 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:56.775760 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:56.776141 (+ 381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:56.776761 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:56.776770 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:56.777290 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:56.777864 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.777879 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.778906 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.778909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.780320 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.780329 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.781064 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.781067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.781269 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.782026 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:56.782046 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:56.784713 (+ 2667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:56.787064 (+ 2351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:56.787070 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:56.787088 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:56.787442 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:56.787744 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:56.787747 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:56.787749 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:56.787912 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:56.788352 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:56.788359 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:56.788648 (+ 289us) client_negotiation.cc:769] Sending connection context
0619 18:42:56.789657 (+ 1009us) client_negotiation.cc:240] Negotiation successful
0619 18:42:56.789795 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":98,"spinlock_wait_cycles":2304,"thread_start_us":114,"threads_started":1}
I20260619 18:42:56.791608 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.773804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:57347 (local address 127.21.143.125:39923)
0619 18:42:56.773966 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:56.773971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:56.774927 (+ 956us) server_negotiation.cc:407] Connection header received
0619 18:42:56.776402 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:56.776405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:56.776460 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:56.776519 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:56.778020 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.778560 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.780448 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.780793 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.780868 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.785306 (+ 4438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:56.785311 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:56.785316 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:56.785364 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:56.787375 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.787883 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.787886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.787888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.787967 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.788263 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.788267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.788269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.788496 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:56.788591 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:56.791344 (+ 2753us) server_negotiation.cc:299] Negotiation successful
0619 18:42:56.791466 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 18:42:56.792557 29425 heartbeater.cc:347] Connected to a master server at 127.21.143.125:39923
I20260619 18:42:56.792557 29423 heartbeater.cc:347] Connected to a master server at 127.21.143.124:37717
I20260619 18:42:56.792825 29423 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:56.792827 29425 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:56.792940 29424 heartbeater.cc:347] Connected to a master server at 127.21.143.126:39877
I20260619 18:42:56.793037 29424 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:56.793457 29423 heartbeater.cc:511] Master 127.21.143.124:37717 requested a full tablet report, sending...
I20260619 18:42:56.793458 29424 heartbeater.cc:511] Master 127.21.143.126:39877 requested a full tablet report, sending...
I20260619 18:42:56.793458 29425 heartbeater.cc:511] Master 127.21.143.125:39923 requested a full tablet report, sending...
I20260619 18:42:56.794605 28971 ts_manager.cc:194] Registered new tserver with Master: 0928d174307d4be2bf5124d798582bf0 (127.21.143.67:33201)
I20260619 18:42:56.794605 28787 ts_manager.cc:194] Registered new tserver with Master: 0928d174307d4be2bf5124d798582bf0 (127.21.143.67:33201)
I20260619 18:42:56.794618 28185 ts_manager.cc:194] Re-registered known tserver with Master: 0928d174307d4be2bf5124d798582bf0 (127.21.143.67:33201)
I20260619 18:42:56.795447 28185 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.143.67@KRBTEST.COM'} at 127.21.143.67:41997
I20260619 18:42:56.802410 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.792015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50664 (local address 127.21.143.125:39923)
0619 18:42:56.792643 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:56.792647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:56.792899 (+ 252us) server_negotiation.cc:407] Connection header received
0619 18:42:56.793657 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:56.793661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:56.793723 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:56.793772 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:56.794814 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.795639 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.796895 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.797219 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.797291 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.799054 (+ 1763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:56.799058 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:56.799061 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:56.799086 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:56.800550 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.801034 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.801038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.801040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.801119 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.801502 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.801506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.801508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.801683 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:56.801832 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:56.802068 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 18:42:56.802201 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549}
I20260619 18:42:56.804358 29157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.792044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60468 (local address 127.21.143.124:37717)
0619 18:42:56.792178 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:56.792184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:56.795628 (+ 3444us) server_negotiation.cc:407] Connection header received
0619 18:42:56.795753 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:56.795756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:56.795798 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:56.795848 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:56.796787 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.797374 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.798411 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.798624 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.798739 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.800197 (+ 1458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:56.800200 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:56.800203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:56.800233 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:56.802273 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.802966 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.802971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.802972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.803063 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.803436 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.803439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.803441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.803672 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:56.803813 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:56.804040 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 18:42:56.804158 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 18:42:56.806068 28224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.792266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60930 (local address 127.21.143.126:39877)
0619 18:42:56.793001 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:56.793005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:56.796932 (+ 3927us) server_negotiation.cc:407] Connection header received
0619 18:42:56.797070 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:56.797073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:56.797111 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:56.797156 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:56.798074 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.798907 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.800696 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.801113 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.801216 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.802722 (+ 1506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:56.802732 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:56.802735 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:56.802765 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:56.804421 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.804862 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.804866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.804868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.804961 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.805225 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.805228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.805232 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.805412 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:56.805550 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:56.805781 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 18:42:56.805903 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648}
I20260619 18:42:56.808882 28185 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:42:56.812163 28185 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:42:56.814924 29438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "81d2c79c01de40299473b15145e1bac9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: false } } peers { permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: NON_VOTER last_known_addr { host: "127.21.143.124" port: 37717 } attrs { promote: true } } }
I20260619 18:42:56.816823 29438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:56.818326 28744 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9: attempting to promote NON_VOTER dab52bca615747c89249a864e1553522 to VOTER
I20260619 18:42:56.818711 28744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: false } } peers { permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 37717 } attrs { promote: false } }
I20260619 18:42:56.819854 28796 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [term 1 FOLLOWER]: Refusing update from remote peer 81d2c79c01de40299473b15145e1bac9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 18:42:56.820132 28981 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [term 1 LEARNER]: Refusing update from remote peer 81d2c79c01de40299473b15145e1bac9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 18:42:56.820338 29446 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60382803ba744b659939734b7e645990" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 18:42:56.820508 29438 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 37717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 18:42:56.821849 29439 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, dab52bca615747c89249a864e1553522 (127.21.143.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: false } } peers { permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 37717 } attrs { promote: false } } }
I20260619 18:42:56.822212 28796 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, dab52bca615747c89249a864e1553522 (127.21.143.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: false } } peers { permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 37717 } attrs { promote: false } } }
I20260619 18:42:56.823449 29439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "81d2c79c01de40299473b15145e1bac9" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: false } } peers { permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 37717 } attrs { promote: false } } }
I20260619 18:42:56.823555 29439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:56.823684 29439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "81d2c79c01de40299473b15145e1bac9" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: false } } peers { permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 37717 } attrs { promote: false } } }
I20260619 18:42:56.823773 29439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:56.823937 29448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "81d2c79c01de40299473b15145e1bac9" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: false } } peers { permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 37717 } attrs { promote: false } } }
I20260619 18:42:56.824179 29439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "81d2c79c01de40299473b15145e1bac9" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: false } } peers { permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 37717 } attrs { promote: false } } }
I20260619 18:42:56.824278 29439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:56.824452 29448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:42:56.824741 28981 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, dab52bca615747c89249a864e1553522 (127.21.143.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: false } } peers { permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 37717 } attrs { promote: false } } }
I20260619 18:42:56.826531 29010 sys_catalog.cc:455] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "81d2c79c01de40299473b15145e1bac9" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: false } } peers { permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 37717 } attrs { promote: false } } }
I20260619 18:42:56.826666 29010 sys_catalog.cc:458] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:42:56.833735 28742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.827568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:44621 (local address 127.0.0.1:41798)
0619 18:42:56.827699 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:56.827725 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:56.827809 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:56.830248 (+ 2439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:56.830252 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:56.830270 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:56.830384 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.830393 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.831724 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.831729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.832538 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.832547 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.833338 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.833341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.833424 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.833452 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 18:42:56.833514 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 18:42:56.833584 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 18:42:56.834087 29449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.828009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41798 (local address 127.21.143.65:44621)
0619 18:42:56.829748 (+ 1739us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:56.829763 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:56.829783 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:42:56.829852 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:56.829857 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:56.830104 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:56.830281 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:56.830512 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.831610 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.832655 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.833216 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.833341 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.833562 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:56.833612 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 18:42:56.833700 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1625,"thread_start_us":91,"threads_started":1}
I20260619 18:42:56.835628 29450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.828159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:38767 (local address 127.0.0.1:33580)
0619 18:42:56.830512 (+ 2353us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:56.830526 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:56.830638 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:56.831269 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:56.831273 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:56.831289 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:56.831429 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.831435 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.832845 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.832848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.834146 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.834157 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.835251 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.835254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.835374 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.835397 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 18:42:56.835461 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 18:42:56.835514 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2283,"mutex_wait_us":4,"thread_start_us":139,"threads_started":1}
I20260619 18:42:56.835767 29453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.828988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33580 (local address 127.21.143.66:38767)
0619 18:42:56.829295 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:56.829307 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:56.830625 (+ 1318us) server_negotiation.cc:407] Connection header received
0619 18:42:56.830753 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:56.830761 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:56.830972 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:56.831117 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:56.831552 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.832717 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.834298 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.835136 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.835253 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.835402 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:56.835520 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 18:42:56.835628 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":96,"threads_started":1}
I20260619 18:42:56.836833 29452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.828600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51360 (local address 127.21.143.67:33201)
0619 18:42:56.828905 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:56.828915 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:56.830910 (+ 1995us) server_negotiation.cc:407] Connection header received
0619 18:42:56.831115 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:56.831120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:56.831284 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:56.831421 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:56.831822 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.832899 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.834309 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.835136 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.835229 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.835402 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:56.836558 (+ 1156us) server_negotiation.cc:299] Negotiation successful
0619 18:42:56.836698 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":110,"threads_started":1}
I20260619 18:42:56.838559 29084 tablet_service.cc:1467] Processing CreateTablet for tablet ee490a8240e349d39a8b768572137dd3 (DEFAULT_TABLE table=test-table [id=65cc5e00f28149b7af440e40fb74c60c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:42:56.838560 29221 tablet_service.cc:1467] Processing CreateTablet for tablet ee490a8240e349d39a8b768572137dd3 (DEFAULT_TABLE table=test-table [id=65cc5e00f28149b7af440e40fb74c60c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:42:56.839426 29451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.828181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:33201 (local address 127.0.0.1:51360)
0619 18:42:56.830828 (+ 2647us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:56.830840 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:56.830971 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:56.831595 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:56.831600 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:56.831618 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:56.831687 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.831692 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.833022 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.833025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.834184 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.834194 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.836314 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.836317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.836404 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.836427 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 18:42:56.836503 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 18:42:56.836573 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2591,"mutex_wait_us":27,"thread_start_us":128,"threads_started":1}
I20260619 18:42:56.839792 29221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee490a8240e349d39a8b768572137dd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:56.840481 29084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee490a8240e349d39a8b768572137dd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:56.841894 29357 tablet_service.cc:1467] Processing CreateTablet for tablet ee490a8240e349d39a8b768572137dd3 (DEFAULT_TABLE table=test-table [id=65cc5e00f28149b7af440e40fb74c60c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:42:56.842818 29357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee490a8240e349d39a8b768572137dd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:56.849450 29454 tablet_bootstrap.cc:492] T ee490a8240e349d39a8b768572137dd3 P 053fb01d91d342f1af627e19f3e04f53: Bootstrap starting.
I20260619 18:42:56.849637 29456 tablet_bootstrap.cc:492] T ee490a8240e349d39a8b768572137dd3 P 0928d174307d4be2bf5124d798582bf0: Bootstrap starting.
I20260619 18:42:56.851426 29456 tablet_bootstrap.cc:654] T ee490a8240e349d39a8b768572137dd3 P 0928d174307d4be2bf5124d798582bf0: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:56.851426 29454 tablet_bootstrap.cc:654] T ee490a8240e349d39a8b768572137dd3 P 053fb01d91d342f1af627e19f3e04f53: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:56.851579 29455 tablet_bootstrap.cc:492] T ee490a8240e349d39a8b768572137dd3 P 580c2c8aeac04a6899c26151d90802ca: Bootstrap starting.
I20260619 18:42:56.852202 29456 log.cc:824] T ee490a8240e349d39a8b768572137dd3 P 0928d174307d4be2bf5124d798582bf0: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:56.852293 29454 log.cc:824] T ee490a8240e349d39a8b768572137dd3 P 053fb01d91d342f1af627e19f3e04f53: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:56.853464 29455 tablet_bootstrap.cc:654] T ee490a8240e349d39a8b768572137dd3 P 580c2c8aeac04a6899c26151d90802ca: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:56.854149 29456 tablet_bootstrap.cc:492] T ee490a8240e349d39a8b768572137dd3 P 0928d174307d4be2bf5124d798582bf0: No bootstrap required, opened a new log
I20260619 18:42:56.854362 29456 ts_tablet_manager.cc:1397] T ee490a8240e349d39a8b768572137dd3 P 0928d174307d4be2bf5124d798582bf0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 18:42:56.854632 29455 log.cc:824] T ee490a8240e349d39a8b768572137dd3 P 580c2c8aeac04a6899c26151d90802ca: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:56.854970 29454 tablet_bootstrap.cc:492] T ee490a8240e349d39a8b768572137dd3 P 053fb01d91d342f1af627e19f3e04f53: No bootstrap required, opened a new log
I20260619 18:42:56.855155 29454 ts_tablet_manager.cc:1397] T ee490a8240e349d39a8b768572137dd3 P 053fb01d91d342f1af627e19f3e04f53: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260619 18:42:56.856565 29455 tablet_bootstrap.cc:492] T ee490a8240e349d39a8b768572137dd3 P 580c2c8aeac04a6899c26151d90802ca: No bootstrap required, opened a new log
I20260619 18:42:56.856786 29455 ts_tablet_manager.cc:1397] T ee490a8240e349d39a8b768572137dd3 P 580c2c8aeac04a6899c26151d90802ca: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 18:42:56.858114 29456 raft_consensus.cc:348] T ee490a8240e349d39a8b768572137dd3 P 0928d174307d4be2bf5124d798582bf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "053fb01d91d342f1af627e19f3e04f53" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44621 } } peers { permanent_uuid: "0928d174307d4be2bf5124d798582bf0" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33201 } } peers { permanent_uuid: "580c2c8aeac04a6899c26151d90802ca" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 38767 } }
I20260619 18:42:56.858378 29456 raft_consensus.cc:374] T ee490a8240e349d39a8b768572137dd3 P 0928d174307d4be2bf5124d798582bf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:56.858440 29456 raft_consensus.cc:729] T ee490a8240e349d39a8b768572137dd3 P 0928d174307d4be2bf5124d798582bf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0928d174307d4be2bf5124d798582bf0, State: Initialized, Role: FOLLOWER
I20260619 18:42:56.858887 29456 consensus_queue.cc:260] T ee490a8240e349d39a8b768572137dd3 P 0928d174307d4be2bf5124d798582bf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "053fb01d91d342f1af627e19f3e04f53" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44621 } } peers { permanent_uuid: "0928d174307d4be2bf5124d798582bf0" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33201 } } peers { permanent_uuid: "580c2c8aeac04a6899c26151d90802ca" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 38767 } }
I20260619 18:42:56.859293 29454 raft_consensus.cc:348] T ee490a8240e349d39a8b768572137dd3 P 053fb01d91d342f1af627e19f3e04f53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "053fb01d91d342f1af627e19f3e04f53" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44621 } } peers { permanent_uuid: "0928d174307d4be2bf5124d798582bf0" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33201 } } peers { permanent_uuid: "580c2c8aeac04a6899c26151d90802ca" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 38767 } }
I20260619 18:42:56.859553 29454 raft_consensus.cc:374] T ee490a8240e349d39a8b768572137dd3 P 053fb01d91d342f1af627e19f3e04f53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:56.859603 29454 raft_consensus.cc:729] T ee490a8240e349d39a8b768572137dd3 P 053fb01d91d342f1af627e19f3e04f53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 053fb01d91d342f1af627e19f3e04f53, State: Initialized, Role: FOLLOWER
I20260619 18:42:56.859965 29456 ts_tablet_manager.cc:1428] T ee490a8240e349d39a8b768572137dd3 P 0928d174307d4be2bf5124d798582bf0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 18:42:56.860046 29454 consensus_queue.cc:260] T ee490a8240e349d39a8b768572137dd3 P 053fb01d91d342f1af627e19f3e04f53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "053fb01d91d342f1af627e19f3e04f53" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44621 } } peers { permanent_uuid: "0928d174307d4be2bf5124d798582bf0" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33201 } } peers { permanent_uuid: "580c2c8aeac04a6899c26151d90802ca" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 38767 } }
I20260619 18:42:56.860951 29454 ts_tablet_manager.cc:1428] T ee490a8240e349d39a8b768572137dd3 P 053fb01d91d342f1af627e19f3e04f53: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 18:42:56.861476 29151 heartbeater.cc:503] Master 127.21.143.126:39877 was elected leader, sending a full tablet report...
I20260619 18:42:56.861789 29424 heartbeater.cc:503] Master 127.21.143.126:39877 was elected leader, sending a full tablet report...
I20260619 18:42:56.866407 29461 raft_consensus.cc:482] T ee490a8240e349d39a8b768572137dd3 P 053fb01d91d342f1af627e19f3e04f53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:42:56.866573 29461 raft_consensus.cc:504] T ee490a8240e349d39a8b768572137dd3 P 053fb01d91d342f1af627e19f3e04f53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "053fb01d91d342f1af627e19f3e04f53" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44621 } } peers { permanent_uuid: "0928d174307d4be2bf5124d798582bf0" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33201 } } peers { permanent_uuid: "580c2c8aeac04a6899c26151d90802ca" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 38767 } }
I20260619 18:42:56.867719 29461 leader_election.cc:288] T ee490a8240e349d39a8b768572137dd3 P 053fb01d91d342f1af627e19f3e04f53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0928d174307d4be2bf5124d798582bf0 (127.21.143.67:33201), 580c2c8aeac04a6899c26151d90802ca (127.21.143.66:38767)
I20260619 18:42:56.869448 29455 raft_consensus.cc:348] T ee490a8240e349d39a8b768572137dd3 P 580c2c8aeac04a6899c26151d90802ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "053fb01d91d342f1af627e19f3e04f53" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44621 } } peers { permanent_uuid: "0928d174307d4be2bf5124d798582bf0" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33201 } } peers { permanent_uuid: "580c2c8aeac04a6899c26151d90802ca" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 38767 } }
I20260619 18:42:56.869696 29455 raft_consensus.cc:374] T ee490a8240e349d39a8b768572137dd3 P 580c2c8aeac04a6899c26151d90802ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:56.869753 29455 raft_consensus.cc:729] T ee490a8240e349d39a8b768572137dd3 P 580c2c8aeac04a6899c26151d90802ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 580c2c8aeac04a6899c26151d90802ca, State: Initialized, Role: FOLLOWER
I20260619 18:42:56.870208 29455 consensus_queue.cc:260] T ee490a8240e349d39a8b768572137dd3 P 580c2c8aeac04a6899c26151d90802ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "053fb01d91d342f1af627e19f3e04f53" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44621 } } peers { permanent_uuid: "0928d174307d4be2bf5124d798582bf0" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33201 } } peers { permanent_uuid: "580c2c8aeac04a6899c26151d90802ca" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 38767 } }
I20260619 18:42:56.870842 29288 heartbeater.cc:503] Master 127.21.143.126:39877 was elected leader, sending a full tablet report...
I20260619 18:42:56.871233 29455 ts_tablet_manager.cc:1428] T ee490a8240e349d39a8b768572137dd3 P 580c2c8aeac04a6899c26151d90802ca: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260619 18:42:56.872421 29452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.868108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:53109 (local address 127.21.143.67:33201)
0619 18:42:56.868209 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:56.868214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:56.868376 (+ 162us) server_negotiation.cc:407] Connection header received
0619 18:42:56.868516 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:56.868519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:56.868563 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:56.868640 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:56.868905 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.869659 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.871132 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.871730 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.871822 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.871875 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:56.872210 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 18:42:56.872294 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 18:42:56.872766 29462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.867830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:33201 (local address 127.21.143.65:53109)
0619 18:42:56.868270 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:56.868286 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:56.868406 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:56.868667 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:56.868673 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:56.868692 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:56.868806 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.868812 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.869795 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.869798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.870995 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.871004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.871967 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.871971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.872066 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.872091 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 18:42:56.872542 (+ 451us) client_negotiation.cc:240] Negotiation successful
0619 18:42:56.872623 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":70,"threads_started":1}
I20260619 18:42:56.873325 29377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee490a8240e349d39a8b768572137dd3" candidate_uuid: "053fb01d91d342f1af627e19f3e04f53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0928d174307d4be2bf5124d798582bf0" is_pre_election: true
I20260619 18:42:56.873602 29377 raft_consensus.cc:2454] T ee490a8240e349d39a8b768572137dd3 P 0928d174307d4be2bf5124d798582bf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 053fb01d91d342f1af627e19f3e04f53 in term 0.
I20260619 18:42:56.874037 29039 leader_election.cc:302] T ee490a8240e349d39a8b768572137dd3 P 053fb01d91d342f1af627e19f3e04f53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 053fb01d91d342f1af627e19f3e04f53, 0928d174307d4be2bf5124d798582bf0; no voters:
I20260619 18:42:56.874480 29461 raft_consensus.cc:2781] T ee490a8240e349d39a8b768572137dd3 P 053fb01d91d342f1af627e19f3e04f53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:42:56.874619 29461 raft_consensus.cc:482] T ee490a8240e349d39a8b768572137dd3 P 053fb01d91d342f1af627e19f3e04f53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:42:56.874737 29461 raft_consensus.cc:3037] T ee490a8240e349d39a8b768572137dd3 P 053fb01d91d342f1af627e19f3e04f53 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:56.876273 29461 raft_consensus.cc:504] T ee490a8240e349d39a8b768572137dd3 P 053fb01d91d342f1af627e19f3e04f53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "053fb01d91d342f1af627e19f3e04f53" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44621 } } peers { permanent_uuid: "0928d174307d4be2bf5124d798582bf0" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33201 } } peers { permanent_uuid: "580c2c8aeac04a6899c26151d90802ca" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 38767 } }
I20260619 18:42:56.876726 29461 leader_election.cc:288] T ee490a8240e349d39a8b768572137dd3 P 053fb01d91d342f1af627e19f3e04f53 [CANDIDATE]: Term 1 election: Requested vote from peers 0928d174307d4be2bf5124d798582bf0 (127.21.143.67:33201), 580c2c8aeac04a6899c26151d90802ca (127.21.143.66:38767)
I20260619 18:42:56.876960 29377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee490a8240e349d39a8b768572137dd3" candidate_uuid: "053fb01d91d342f1af627e19f3e04f53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0928d174307d4be2bf5124d798582bf0"
I20260619 18:42:56.877096 29377 raft_consensus.cc:3037] T ee490a8240e349d39a8b768572137dd3 P 0928d174307d4be2bf5124d798582bf0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:56.877775 29453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.868012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:42183 (local address 127.21.143.66:38767)
0619 18:42:56.868143 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:56.868149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:56.872988 (+ 4839us) server_negotiation.cc:407] Connection header received
0619 18:42:56.873167 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:56.873170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:56.873213 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:56.873260 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:56.873868 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.874629 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.876481 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.877098 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.877202 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.877250 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:56.877572 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 18:42:56.877646 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 18:42:56.878091 29462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.867935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:38767 (local address 127.21.143.65:42183)
0619 18:42:56.872923 (+ 4988us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:56.872938 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:56.873051 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:56.873654 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:56.873657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:56.873674 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:56.873757 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.873762 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.875153 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.875156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.876351 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:56.876362 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.877355 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.877358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:56.877446 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.877468 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 18:42:56.877898 (+ 430us) client_negotiation.cc:240] Negotiation successful
0619 18:42:56.877973 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4922,"thread_start_us":65,"threads_started":1}
I20260619 18:42:56.878707 29377 raft_consensus.cc:2454] T ee490a8240e349d39a8b768572137dd3 P 0928d174307d4be2bf5124d798582bf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 053fb01d91d342f1af627e19f3e04f53 in term 1.
I20260619 18:42:56.878993 29039 leader_election.cc:302] T ee490a8240e349d39a8b768572137dd3 P 053fb01d91d342f1af627e19f3e04f53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 053fb01d91d342f1af627e19f3e04f53, 0928d174307d4be2bf5124d798582bf0; no voters:
I20260619 18:42:56.879165 29461 raft_consensus.cc:2781] T ee490a8240e349d39a8b768572137dd3 P 053fb01d91d342f1af627e19f3e04f53 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:56.879352 29461 raft_consensus.cc:686] T ee490a8240e349d39a8b768572137dd3 P 053fb01d91d342f1af627e19f3e04f53 [term 1 LEADER]: Becoming Leader. State: Replica: 053fb01d91d342f1af627e19f3e04f53, State: Running, Role: LEADER
I20260619 18:42:56.879470 29240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee490a8240e349d39a8b768572137dd3" candidate_uuid: "053fb01d91d342f1af627e19f3e04f53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "580c2c8aeac04a6899c26151d90802ca"
I20260619 18:42:56.879716 29240 raft_consensus.cc:3037] T ee490a8240e349d39a8b768572137dd3 P 580c2c8aeac04a6899c26151d90802ca [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:56.879688 29461 consensus_queue.cc:237] T ee490a8240e349d39a8b768572137dd3 P 053fb01d91d342f1af627e19f3e04f53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "053fb01d91d342f1af627e19f3e04f53" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44621 } } peers { permanent_uuid: "0928d174307d4be2bf5124d798582bf0" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33201 } } peers { permanent_uuid: "580c2c8aeac04a6899c26151d90802ca" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 38767 } }
I20260619 18:42:56.880909 29240 raft_consensus.cc:2454] T ee490a8240e349d39a8b768572137dd3 P 580c2c8aeac04a6899c26151d90802ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 053fb01d91d342f1af627e19f3e04f53 in term 1.
I20260619 18:42:56.879470 29241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee490a8240e349d39a8b768572137dd3" candidate_uuid: "053fb01d91d342f1af627e19f3e04f53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "580c2c8aeac04a6899c26151d90802ca" is_pre_election: true
I20260619 18:42:56.881291 29241 raft_consensus.cc:2362] T ee490a8240e349d39a8b768572137dd3 P 580c2c8aeac04a6899c26151d90802ca [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 053fb01d91d342f1af627e19f3e04f53 in term 1. Re-sending same reply.
I20260619 18:42:56.883714 28183 catalog_manager.cc:5495] T ee490a8240e349d39a8b768572137dd3 P 053fb01d91d342f1af627e19f3e04f53 reported cstate change: term changed from 0 to 1, leader changed from <none> to 053fb01d91d342f1af627e19f3e04f53 (127.21.143.65). New cstate: current_term: 1 leader_uuid: "053fb01d91d342f1af627e19f3e04f53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "053fb01d91d342f1af627e19f3e04f53" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0928d174307d4be2bf5124d798582bf0" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "580c2c8aeac04a6899c26151d90802ca" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 38767 } health_report { overall_health: UNKNOWN } } }
I20260619 18:42:56.911661 29449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.908420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41800 (local address 127.21.143.65:44621)
0619 18:42:56.908540 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:56.908543 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:56.908556 (+ 13us) server_negotiation.cc:407] Connection header received
0619 18:42:56.908657 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:56.908659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:56.908712 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:56.908804 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:42:56.908985 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.909489 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.910271 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.910477 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.910554 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.910810 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:42:56.911252 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:42:56.911416 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:56.911447 (+ 31us) server_negotiation.cc:299] Negotiation successful
0619 18:42:56.911499 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 18:42:56.917550 29377 raft_consensus.cc:1261] T ee490a8240e349d39a8b768572137dd3 P 0928d174307d4be2bf5124d798582bf0 [term 1 FOLLOWER]: Refusing update from remote peer 053fb01d91d342f1af627e19f3e04f53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:42:56.917550 29241 raft_consensus.cc:1261] T ee490a8240e349d39a8b768572137dd3 P 580c2c8aeac04a6899c26151d90802ca [term 1 FOLLOWER]: Refusing update from remote peer 053fb01d91d342f1af627e19f3e04f53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:42:56.918236 29461 consensus_queue.cc:1035] T ee490a8240e349d39a8b768572137dd3 P 053fb01d91d342f1af627e19f3e04f53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "580c2c8aeac04a6899c26151d90802ca" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 38767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:42:56.918438 29465 consensus_queue.cc:1035] T ee490a8240e349d39a8b768572137dd3 P 053fb01d91d342f1af627e19f3e04f53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0928d174307d4be2bf5124d798582bf0" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:42:56.930526 29470 mvcc.cc:205] Tried to move back new op lower bound from 7298640187046715392 to 7298640186902609920. Current Snapshot: MvccSnapshot[applied={T|T < 7298640187046715392}]
I20260619 18:42:56.939412 28185 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:42:56.939615 28185 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:42:56.944051 28185 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81d2c79c01de40299473b15145e1bac9: Sending DeleteTablet for 3 replicas of tablet ee490a8240e349d39a8b768572137dd3
I20260619 18:42:56.945384 29084 tablet_service.cc:1514] Processing DeleteTablet for tablet ee490a8240e349d39a8b768572137dd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:42:56 UTC) from {username='oryx', principal='oryx/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:41798
I20260619 18:42:56.945472 29221 tablet_service.cc:1514] Processing DeleteTablet for tablet ee490a8240e349d39a8b768572137dd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:42:56 UTC) from {username='oryx', principal='oryx/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:33580
I20260619 18:42:56.945463 29357 tablet_service.cc:1514] Processing DeleteTablet for tablet ee490a8240e349d39a8b768572137dd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:42:56 UTC) from {username='oryx', principal='oryx/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:51360
I20260619 18:42:56.946235 29480 tablet_replica.cc:330] T ee490a8240e349d39a8b768572137dd3 P 580c2c8aeac04a6899c26151d90802ca: stopping tablet replica
I20260619 18:42:56.946593 29480 raft_consensus.cc:2229] T ee490a8240e349d39a8b768572137dd3 P 580c2c8aeac04a6899c26151d90802ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:42:56.946983 29480 raft_consensus.cc:2258] T ee490a8240e349d39a8b768572137dd3 P 580c2c8aeac04a6899c26151d90802ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:42:56.947817 29479 tablet_replica.cc:330] T ee490a8240e349d39a8b768572137dd3 P 053fb01d91d342f1af627e19f3e04f53: stopping tablet replica
I20260619 18:42:56.947944 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 28153
I20260619 18:42:56.948582 29479 raft_consensus.cc:2229] T ee490a8240e349d39a8b768572137dd3 P 053fb01d91d342f1af627e19f3e04f53 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:42:56.949040 29479 raft_consensus.cc:2258] T ee490a8240e349d39a8b768572137dd3 P 053fb01d91d342f1af627e19f3e04f53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:42:56.950642 29480 ts_tablet_manager.cc:1905] T ee490a8240e349d39a8b768572137dd3 P 580c2c8aeac04a6899c26151d90802ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:42:56.953030 29480 ts_tablet_manager.cc:1918] T ee490a8240e349d39a8b768572137dd3 P 580c2c8aeac04a6899c26151d90802ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:42:56.953147 29480 log.cc:1196] T ee490a8240e349d39a8b768572137dd3 P 580c2c8aeac04a6899c26151d90802ca: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-1/wal/wals/ee490a8240e349d39a8b768572137dd3
I20260619 18:42:56.953492 29480 ts_tablet_manager.cc:1939] T ee490a8240e349d39a8b768572137dd3 P 580c2c8aeac04a6899c26151d90802ca: Deleting consensus metadata
I20260619 18:42:56.955717 29479 ts_tablet_manager.cc:1905] T ee490a8240e349d39a8b768572137dd3 P 053fb01d91d342f1af627e19f3e04f53: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 18:42:56.956362 29176 connection.cc:535] server connection from 127.0.0.1:33580 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 18:42:56.958670 29479 ts_tablet_manager.cc:1918] T ee490a8240e349d39a8b768572137dd3 P 053fb01d91d342f1af627e19f3e04f53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:42:56.958771 29479 log.cc:1196] T ee490a8240e349d39a8b768572137dd3 P 053fb01d91d342f1af627e19f3e04f53: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-0/wal/wals/ee490a8240e349d39a8b768572137dd3
I20260619 18:42:56.959116 29479 ts_tablet_manager.cc:1939] T ee490a8240e349d39a8b768572137dd3 P 053fb01d91d342f1af627e19f3e04f53: Deleting consensus metadata
I20260619 18:42:56.959501 29481 tablet_replica.cc:330] T ee490a8240e349d39a8b768572137dd3 P 0928d174307d4be2bf5124d798582bf0: stopping tablet replica
I20260619 18:42:56.959818 29481 raft_consensus.cc:2229] T ee490a8240e349d39a8b768572137dd3 P 0928d174307d4be2bf5124d798582bf0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:42:56.960129 29481 raft_consensus.cc:2258] T ee490a8240e349d39a8b768572137dd3 P 0928d174307d4be2bf5124d798582bf0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 18:42:56.960212 29039 connection.cc:441] server connection from 127.0.0.1:41798 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41798 (request call id 1) could send its response
I20260619 18:42:56.961874 29481 ts_tablet_manager.cc:1905] T ee490a8240e349d39a8b768572137dd3 P 0928d174307d4be2bf5124d798582bf0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:42:56.967555 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.948023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50674 (local address 127.21.143.125:39923)
0619 18:42:56.948462 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:56.948467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:56.948480 (+ 13us) server_negotiation.cc:407] Connection header received
0619 18:42:56.948518 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:56.948521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:56.948564 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:56.948642 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:56.956673 (+ 8031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.957469 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.960773 (+ 3304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.961113 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.961226 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.963240 (+ 2014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:56.963244 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:56.963247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:56.963273 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:56.965056 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.965536 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.965539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.965541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.965616 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.966292 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.966296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.966298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.966499 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:56.966607 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:56.967192 (+ 585us) server_negotiation.cc:299] Negotiation successful
0619 18:42:56.967396 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367}
I20260619 18:42:56.971125 29488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.960657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60494 (local address 127.21.143.124:37717)
0619 18:42:56.961556 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:56.961561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:56.961574 (+ 13us) server_negotiation.cc:407] Connection header received
0619 18:42:56.961618 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:56.961621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:56.961682 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:56.961778 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:56.962734 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.963607 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.964840 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.965218 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.965315 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.966888 (+ 1573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:56.966891 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:56.966893 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:56.966942 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:56.968922 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.969452 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.969456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.969457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.969547 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.969875 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.969880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.969882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.970116 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:56.970248 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:56.970776 (+ 528us) server_negotiation.cc:299] Negotiation successful
0619 18:42:56.970915 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":806,"thread_start_us":64,"threads_started":1}
I20260619 18:42:56.971750 29490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.961018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50680 (local address 127.21.143.125:39923)
0619 18:42:56.962598 (+ 1580us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:56.962610 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:56.962625 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:42:56.962663 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:56.962669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:56.962719 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:56.962796 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:56.963965 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.964748 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.965864 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.966259 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.966373 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.968154 (+ 1781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:56.968159 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:56.968163 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:56.968189 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:56.969949 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.970400 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.970404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.970406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.970490 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.970771 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.970774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.970776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.970967 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:56.971080 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:56.971444 (+ 364us) server_negotiation.cc:299] Negotiation successful
0619 18:42:56.971562 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1506,"thread_start_us":80,"threads_started":1}
I20260619 18:42:56.976897 29481 ts_tablet_manager.cc:1918] T ee490a8240e349d39a8b768572137dd3 P 0928d174307d4be2bf5124d798582bf0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:42:56.977016 29481 log.cc:1196] T ee490a8240e349d39a8b768572137dd3 P 0928d174307d4be2bf5124d798582bf0: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894542572984-22077-0/minicluster-data/ts-2/wal/wals/ee490a8240e349d39a8b768572137dd3
I20260619 18:42:56.977380 29481 ts_tablet_manager.cc:1939] T ee490a8240e349d39a8b768572137dd3 P 0928d174307d4be2bf5124d798582bf0: Deleting consensus metadata
I20260619 18:42:56.978183 29157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:56.948119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60484 (local address 127.21.143.124:37717)
0619 18:42:56.948887 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:56.948891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:56.969060 (+ 20169us) server_negotiation.cc:407] Connection header received
0619 18:42:56.969228 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:56.969231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:56.969273 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:56.969320 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:56.970004 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.970782 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.973034 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:56.973243 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:56.973319 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:56.974784 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:56.974787 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:56.974790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:56.974814 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:56.976505 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.976944 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.976947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.976948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.977043 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:56.977332 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:56.977335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:56.977337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:56.977563 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:56.977686 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:56.977913 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 18:42:56.978026 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":696}
W20260619 18:42:56.978613 29313 connection.cc:441] server connection from 127.0.0.1:51360 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51360 (request call id 1) could send its response
I20260619 18:42:57.930480 29491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:57.929965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.21.143.66:43651)
0619 18:42:57.930228 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:57.930353 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":83,"threads_started":1}
I20260619 18:42:57.931115 29492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:57.930473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.21.143.67:44447)
0619 18:42:57.930941 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:57.931021 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":63,"threads_started":1}
W20260619 18:42:57.931519 29424 heartbeater.cc:644] Failed to heartbeat to 127.21.143.126:39877 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.143.126:39877: connect: Connection refused (error 111)
W20260619 18:42:57.931520 29288 heartbeater.cc:644] Failed to heartbeat to 127.21.143.126:39877 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.143.126:39877: connect: Connection refused (error 111)
I20260619 18:42:57.933187 29493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:57.932852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.21.143.65:41431)
0619 18:42:57.933053 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:57.933116 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":76,"threads_started":1}
W20260619 18:42:57.933411 29151 heartbeater.cc:644] Failed to heartbeat to 127.21.143.126:39877 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.143.126:39877: connect: Connection refused (error 111)
I20260619 18:42:58.444819 29494 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 81d2c79c01de40299473b15145e1bac9)
I20260619 18:42:58.445055 29495 raft_consensus.cc:482] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 81d2c79c01de40299473b15145e1bac9)
I20260619 18:42:58.445127 29494 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: false } } peers { permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 37717 } attrs { promote: false } }
I20260619 18:42:58.445248 29495 raft_consensus.cc:504] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: false } } peers { permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 37717 } attrs { promote: false } }
I20260619 18:42:58.446251 29494 leader_election.cc:288] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 81d2c79c01de40299473b15145e1bac9 (127.21.143.126:39877), dab52bca615747c89249a864e1553522 (127.21.143.124:37717)
I20260619 18:42:58.446431 29495 leader_election.cc:288] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 81d2c79c01de40299473b15145e1bac9 (127.21.143.126:39877), 60382803ba744b659939734b7e645990 (127.21.143.125:39923)
I20260619 18:42:58.447001 29498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:58.446403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.0.0.1:33158)
0619 18:42:58.446788 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:58.446870 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":83,"threads_started":1}
W20260619 18:42:58.447202 28955 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.143.126:39877: connect: Connection refused (error 111)
I20260619 18:42:58.447400 29497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:58.446838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.0.0.1:33164)
0619 18:42:58.447205 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:58.447263 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":89,"threads_started":1}
W20260619 18:42:58.447570 28771 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.143.126:39877: connect: Connection refused (error 111)
I20260619 18:42:58.448429 29497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:58.448198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.0.0.1:33174)
0619 18:42:58.448285 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:58.448343 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260619 18:42:58.448575 28771 leader_election.cc:334] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 81d2c79c01de40299473b15145e1bac9 (127.21.143.126:39877): Network error: Client connection negotiation failed: client connection to 127.21.143.126:39877: connect: Connection refused (error 111)
I20260619 18:42:58.448705 29498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:58.448403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.0.0.1:33178)
0619 18:42:58.448580 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:58.448629 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260619 18:42:58.448799 28955 leader_election.cc:334] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 81d2c79c01de40299473b15145e1bac9 (127.21.143.126:39877): Network error: Client connection negotiation failed: client connection to 127.21.143.126:39877: connect: Connection refused (error 111)
I20260619 18:42:58.451165 29501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:58.446893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50692 (local address 127.21.143.125:39923)
0619 18:42:58.447444 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:58.447448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:58.447464 (+ 16us) server_negotiation.cc:407] Connection header received
0619 18:42:58.447520 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:58.447527 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:58.447586 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:58.447674 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:58.448101 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:58.448893 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:58.450096 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:58.450600 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:58.450689 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:58.450730 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:58.450956 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 18:42:58.451038 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":58,"threads_started":1}
I20260619 18:42:58.451165 29496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:58.446403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.125:39923 (local address 127.0.0.1:50692)
0619 18:42:58.447108 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:58.447121 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:58.447225 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:58.447805 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:58.447809 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:58.447828 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:58.447967 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:58.447975 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:58.449023 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:58.449026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:58.449964 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:58.449972 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:58.450706 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:58.450710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:58.450827 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:58.450854 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 18:42:58.450936 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 18:42:58.451020 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":168,"spinlock_wait_cycles":14208,"thread_start_us":61,"threads_started":1}
I20260619 18:42:58.451516 29500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:58.446486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:37717 (local address 127.0.0.1:60510)
0619 18:42:58.447050 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:58.447064 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:58.447722 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:58.448139 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:58.448142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:58.448163 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:58.448260 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:58.448266 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:58.449402 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:58.449405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:58.450276 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:58.450285 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:58.451071 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:58.451082 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:58.451165 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:58.451196 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 18:42:58.451276 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 18:42:58.451382 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"spinlock_wait_cycles":2304,"thread_start_us":75,"threads_started":1}
I20260619 18:42:58.451596 29499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:58.446653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60510 (local address 127.21.143.124:37717)
0619 18:42:58.446839 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:58.446844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:58.447659 (+ 815us) server_negotiation.cc:407] Connection header received
0619 18:42:58.447853 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:58.447857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:58.447931 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:58.448025 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:58.448386 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:58.449295 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:58.450392 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:58.450970 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:58.451066 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:58.451108 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:58.451386 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 18:42:58.451467 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":122,"spinlock_wait_cycles":1920,"thread_start_us":51,"threads_started":1}
I20260619 18:42:58.451740 28796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dab52bca615747c89249a864e1553522" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "60382803ba744b659939734b7e645990" is_pre_election: true
I20260619 18:42:58.451993 28796 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dab52bca615747c89249a864e1553522 in term 1.
I20260619 18:42:58.452343 28956 leader_election.cc:302] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60382803ba744b659939734b7e645990, dab52bca615747c89249a864e1553522; no voters: 81d2c79c01de40299473b15145e1bac9
I20260619 18:42:58.452602 29495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 18:42:58.452638 28981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "60382803ba744b659939734b7e645990" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "dab52bca615747c89249a864e1553522" is_pre_election: true
I20260619 18:42:58.452708 29495 raft_consensus.cc:482] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 81d2c79c01de40299473b15145e1bac9)
I20260619 18:42:58.452785 29495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [term 1 FOLLOWER]: Advancing to term 2
I20260619 18:42:58.453869 29495 raft_consensus.cc:504] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: false } } peers { permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 37717 } attrs { promote: false } }
I20260619 18:42:58.454162 28981 raft_consensus.cc:2379] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60382803ba744b659939734b7e645990 in current term 2: Already voted for candidate dab52bca615747c89249a864e1553522 in this term.
I20260619 18:42:58.454317 29495 leader_election.cc:288] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [CANDIDATE]: Term 2 election: Requested vote from peers 81d2c79c01de40299473b15145e1bac9 (127.21.143.126:39877), 60382803ba744b659939734b7e645990 (127.21.143.125:39923)
I20260619 18:42:58.454627 29496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:58.454422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.0.0.1:33192)
0619 18:42:58.454510 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:58.454564 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 18:42:58.454727 28796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dab52bca615747c89249a864e1553522" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "60382803ba744b659939734b7e645990"
I20260619 18:42:58.454856 28796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [term 1 FOLLOWER]: Advancing to term 2
I20260619 18:42:58.455698 28773 leader_election.cc:302] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60382803ba744b659939734b7e645990; no voters: 81d2c79c01de40299473b15145e1bac9, dab52bca615747c89249a864e1553522
I20260619 18:42:58.455998 29496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:58.455693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.0.0.1:33202)
0619 18:42:58.455878 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:58.455937 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 18:42:58.456138 28955 leader_election.cc:334] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 81d2c79c01de40299473b15145e1bac9 (127.21.143.126:39877): Network error: Client connection negotiation failed: client connection to 127.21.143.126:39877: connect: Connection refused (error 111)
I20260619 18:42:58.456176 28796 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dab52bca615747c89249a864e1553522 in term 2.
I20260619 18:42:58.456265 29494 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 18:42:58.456455 28956 leader_election.cc:302] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60382803ba744b659939734b7e645990, dab52bca615747c89249a864e1553522; no voters: 81d2c79c01de40299473b15145e1bac9
I20260619 18:42:58.456646 29495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 18:42:58.456885 29495 raft_consensus.cc:686] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [term 2 LEADER]: Becoming Leader. State: Replica: dab52bca615747c89249a864e1553522, State: Running, Role: LEADER
I20260619 18:42:58.457154 29495 consensus_queue.cc:237] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: false } } peers { permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 37717 } attrs { promote: false } }
I20260619 18:42:58.458403 29504 sys_catalog.cc:455] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dab52bca615747c89249a864e1553522. Latest consensus state: current_term: 2 leader_uuid: "dab52bca615747c89249a864e1553522" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: false } } peers { permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 37717 } attrs { promote: false } } }
I20260619 18:42:58.458536 29504 sys_catalog.cc:458] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:58.458853 29506 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:42:58.461074 29506 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:42:58.461480 29506 catalog_manager.cc:1259] Loaded cluster ID: 9bbe0baa520143faa08c929e05e9b145
I20260619 18:42:58.461529 29506 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:42:58.462100 29506 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:42:58.462491 29506 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522: Loaded TSK: 0
I20260619 18:42:58.463218 29506 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:42:58.901595 29496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:58.901242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.0.0.1:33214)
0619 18:42:58.901419 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:58.901510 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 18:42:58.902310 29496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:58.902117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.0.0.1:33228)
0619 18:42:58.902212 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:58.902259 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 18:42:58.902478 28955 consensus_peers.cc:487] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 -> Peer 81d2c79c01de40299473b15145e1bac9 (127.21.143.126:39877): Couldn't send request to peer 81d2c79c01de40299473b15145e1bac9. Status: Network error: Client connection negotiation failed: client connection to 127.21.143.126:39877: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 18:42:58.932597 29508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:58.932106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.21.143.66:36997)
0619 18:42:58.932431 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:58.932501 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":84,"threads_started":1}
I20260619 18:42:58.932582 29507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:58.932068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.21.143.67:59121)
0619 18:42:58.932389 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:58.932469 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":75,"threads_started":1}
I20260619 18:42:58.934267 29509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:58.933920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.21.143.65:43723)
0619 18:42:58.934126 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:58.934187 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":59,"threads_started":1}
I20260619 18:42:58.950294 28796 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [term 2 FOLLOWER]: Refusing update from remote peer dab52bca615747c89249a864e1553522: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20260619 18:42:58.950917 29504 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60382803ba744b659939734b7e645990" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260619 18:42:58.952168 29494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dab52bca615747c89249a864e1553522. Latest consensus state: current_term: 2 leader_uuid: "dab52bca615747c89249a864e1553522" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: false } } peers { permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 37717 } attrs { promote: false } } }
I20260619 18:42:58.952309 29494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:42:58.953408 29495 sys_catalog.cc:455] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "dab52bca615747c89249a864e1553522" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: false } } peers { permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 37717 } attrs { promote: false } } }
I20260619 18:42:58.953531 29495 sys_catalog.cc:458] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:58.953758 29494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "dab52bca615747c89249a864e1553522" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "81d2c79c01de40299473b15145e1bac9" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39877 } } peers { permanent_uuid: "60382803ba744b659939734b7e645990" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39923 } attrs { promote: false } } peers { permanent_uuid: "dab52bca615747c89249a864e1553522" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 37717 } attrs { promote: false } } }
I20260619 18:42:58.953876 29494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60382803ba744b659939734b7e645990 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:42:59.142899 28970 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:42:59.145972 28970 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:42:59.160753 29520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:59.156264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41804 (local address 127.21.143.65:44621)
0619 18:42:59.156932 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:59.156936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:59.156952 (+ 16us) server_negotiation.cc:407] Connection header received
0619 18:42:59.157000 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:59.157002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:59.157076 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:59.157160 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:59.157392 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:59.158308 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:59.159658 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:59.160104 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:59.160227 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:59.160295 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:59.160492 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 18:42:59.160585 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":62,"threads_started":1}
I20260619 18:42:59.160785 29496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:59.155539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:33201 (local address 127.0.0.1:51366)
0619 18:42:59.155695 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:59.155711 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:59.155806 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:59.156463 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:59.156465 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:59.156485 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:59.156619 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:59.156626 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:59.157696 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:59.157699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:59.158780 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:59.158789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:59.160403 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:59.160407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:59.160516 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:59.160536 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 18:42:59.160604 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 18:42:59.160668 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":21}
I20260619 18:42:59.161093 29516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:59.155667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51366 (local address 127.21.143.67:33201)
0619 18:42:59.156216 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:59.156221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:59.156236 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:42:59.156292 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:59.156299 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:59.156355 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:59.156450 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:59.156744 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:59.157576 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:59.159676 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:59.160285 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:59.160410 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:59.160470 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:59.160634 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 18:42:59.160717 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":79,"threads_started":1}
I20260619 18:42:59.161111 29518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:59.156163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:44621 (local address 127.0.0.1:41804)
0619 18:42:59.156653 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:59.156666 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:59.156764 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:59.157167 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:59.157170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:59.157185 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:59.157274 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:59.157279 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:59.158440 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:59.158444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:59.159538 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:59.159547 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:59.160222 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:59.160226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:59.160303 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:59.160330 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 18:42:59.160403 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 18:42:59.161028 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":24,"thread_start_us":78,"threads_started":1}
I20260619 18:42:59.161250 29519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:59.156181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33584 (local address 127.21.143.66:38767)
0619 18:42:59.156974 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:59.156979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:59.156993 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:42:59.157052 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:59.157055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:59.157099 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:59.157217 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:59.157547 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:59.158379 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:59.159711 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:59.160335 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:59.160425 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:59.160481 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:59.161012 (+ 531us) server_negotiation.cc:299] Negotiation successful
0619 18:42:59.161111 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":64,"threads_started":1}
I20260619 18:42:59.161496 29517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:59.155845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:38767 (local address 127.0.0.1:33584)
0619 18:42:59.156424 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:59.156440 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:59.156535 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:59.157314 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:59.157317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:59.157334 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:59.157412 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:59.157417 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:59.158522 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:59.158526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:59.159588 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:59.159614 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:59.160536 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:59.160540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:59.160665 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:59.160688 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 18:42:59.160754 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 18:42:59.160817 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"spinlock_wait_cycles":768,"thread_start_us":92,"threads_started":1}
I20260619 18:42:59.161644 29357 tablet_service.cc:1467] Processing CreateTablet for tablet 2a6efa0635c243e689b704c218d012f0 (DEFAULT_TABLE table=test-table [id=230e181886b84b54a97c7688113aa4eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:42:59.161959 29357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a6efa0635c243e689b704c218d012f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:59.162257 29221 tablet_service.cc:1467] Processing CreateTablet for tablet 2a6efa0635c243e689b704c218d012f0 (DEFAULT_TABLE table=test-table [id=230e181886b84b54a97c7688113aa4eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:42:59.162580 29221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a6efa0635c243e689b704c218d012f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:59.163744 29084 tablet_service.cc:1467] Processing CreateTablet for tablet 2a6efa0635c243e689b704c218d012f0 (DEFAULT_TABLE table=test-table [id=230e181886b84b54a97c7688113aa4eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:42:59.164021 29084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a6efa0635c243e689b704c218d012f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:59.164619 29521 tablet_bootstrap.cc:492] T 2a6efa0635c243e689b704c218d012f0 P 0928d174307d4be2bf5124d798582bf0: Bootstrap starting.
I20260619 18:42:59.165158 29522 tablet_bootstrap.cc:492] T 2a6efa0635c243e689b704c218d012f0 P 580c2c8aeac04a6899c26151d90802ca: Bootstrap starting.
I20260619 18:42:59.165668 29521 tablet_bootstrap.cc:654] T 2a6efa0635c243e689b704c218d012f0 P 0928d174307d4be2bf5124d798582bf0: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:59.166097 29522 tablet_bootstrap.cc:654] T 2a6efa0635c243e689b704c218d012f0 P 580c2c8aeac04a6899c26151d90802ca: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:59.166863 29523 tablet_bootstrap.cc:492] T 2a6efa0635c243e689b704c218d012f0 P 053fb01d91d342f1af627e19f3e04f53: Bootstrap starting.
I20260619 18:42:59.167676 29523 tablet_bootstrap.cc:654] T 2a6efa0635c243e689b704c218d012f0 P 053fb01d91d342f1af627e19f3e04f53: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:59.168149 29521 tablet_bootstrap.cc:492] T 2a6efa0635c243e689b704c218d012f0 P 0928d174307d4be2bf5124d798582bf0: No bootstrap required, opened a new log
I20260619 18:42:59.168258 29521 ts_tablet_manager.cc:1397] T 2a6efa0635c243e689b704c218d012f0 P 0928d174307d4be2bf5124d798582bf0: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 18:42:59.168424 29522 tablet_bootstrap.cc:492] T 2a6efa0635c243e689b704c218d012f0 P 580c2c8aeac04a6899c26151d90802ca: No bootstrap required, opened a new log
I20260619 18:42:59.168495 29522 ts_tablet_manager.cc:1397] T 2a6efa0635c243e689b704c218d012f0 P 580c2c8aeac04a6899c26151d90802ca: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 18:42:59.168697 29521 raft_consensus.cc:348] T 2a6efa0635c243e689b704c218d012f0 P 0928d174307d4be2bf5124d798582bf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0928d174307d4be2bf5124d798582bf0" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33201 } } peers { permanent_uuid: "580c2c8aeac04a6899c26151d90802ca" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 38767 } } peers { permanent_uuid: "053fb01d91d342f1af627e19f3e04f53" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44621 } }
I20260619 18:42:59.168813 29521 raft_consensus.cc:374] T 2a6efa0635c243e689b704c218d012f0 P 0928d174307d4be2bf5124d798582bf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:59.168857 29521 raft_consensus.cc:729] T 2a6efa0635c243e689b704c218d012f0 P 0928d174307d4be2bf5124d798582bf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0928d174307d4be2bf5124d798582bf0, State: Initialized, Role: FOLLOWER
I20260619 18:42:59.168912 29522 raft_consensus.cc:348] T 2a6efa0635c243e689b704c218d012f0 P 580c2c8aeac04a6899c26151d90802ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0928d174307d4be2bf5124d798582bf0" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33201 } } peers { permanent_uuid: "580c2c8aeac04a6899c26151d90802ca" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 38767 } } peers { permanent_uuid: "053fb01d91d342f1af627e19f3e04f53" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44621 } }
I20260619 18:42:59.168987 29521 consensus_queue.cc:260] T 2a6efa0635c243e689b704c218d012f0 P 0928d174307d4be2bf5124d798582bf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0928d174307d4be2bf5124d798582bf0" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33201 } } peers { permanent_uuid: "580c2c8aeac04a6899c26151d90802ca" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 38767 } } peers { permanent_uuid: "053fb01d91d342f1af627e19f3e04f53" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44621 } }
I20260619 18:42:59.169049 29522 raft_consensus.cc:374] T 2a6efa0635c243e689b704c218d012f0 P 580c2c8aeac04a6899c26151d90802ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:59.169106 29522 raft_consensus.cc:729] T 2a6efa0635c243e689b704c218d012f0 P 580c2c8aeac04a6899c26151d90802ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 580c2c8aeac04a6899c26151d90802ca, State: Initialized, Role: FOLLOWER
I20260619 18:42:59.169238 29522 consensus_queue.cc:260] T 2a6efa0635c243e689b704c218d012f0 P 580c2c8aeac04a6899c26151d90802ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0928d174307d4be2bf5124d798582bf0" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33201 } } peers { permanent_uuid: "580c2c8aeac04a6899c26151d90802ca" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 38767 } } peers { permanent_uuid: "053fb01d91d342f1af627e19f3e04f53" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44621 } }
I20260619 18:42:59.169350 29521 ts_tablet_manager.cc:1428] T 2a6efa0635c243e689b704c218d012f0 P 0928d174307d4be2bf5124d798582bf0: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 18:42:59.169384 29423 heartbeater.cc:503] Master 127.21.143.124:37717 was elected leader, sending a full tablet report...
I20260619 18:42:59.169589 29522 ts_tablet_manager.cc:1428] T 2a6efa0635c243e689b704c218d012f0 P 580c2c8aeac04a6899c26151d90802ca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:42:59.169755 29287 heartbeater.cc:503] Master 127.21.143.124:37717 was elected leader, sending a full tablet report...
I20260619 18:42:59.170272 29507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:59.169787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.21.143.67:53311)
0619 18:42:59.170145 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:59.170202 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286}
I20260619 18:42:59.170274 29508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:59.169977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.21.143.66:50211)
0619 18:42:59.170145 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:59.170203 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260619 18:42:59.171060 29507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:59.170838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.21.143.67:52867)
0619 18:42:59.170975 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:59.171017 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 18:42:59.171897 29508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:59.171646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.21.143.66:54723)
0619 18:42:59.171765 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:59.171838 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 18:42:59.172772 29523 tablet_bootstrap.cc:492] T 2a6efa0635c243e689b704c218d012f0 P 053fb01d91d342f1af627e19f3e04f53: No bootstrap required, opened a new log
I20260619 18:42:59.172865 29523 ts_tablet_manager.cc:1397] T 2a6efa0635c243e689b704c218d012f0 P 053fb01d91d342f1af627e19f3e04f53: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260619 18:42:59.173316 29523 raft_consensus.cc:348] T 2a6efa0635c243e689b704c218d012f0 P 053fb01d91d342f1af627e19f3e04f53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0928d174307d4be2bf5124d798582bf0" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33201 } } peers { permanent_uuid: "580c2c8aeac04a6899c26151d90802ca" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 38767 } } peers { permanent_uuid: "053fb01d91d342f1af627e19f3e04f53" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44621 } }
I20260619 18:42:59.173429 29523 raft_consensus.cc:374] T 2a6efa0635c243e689b704c218d012f0 P 053fb01d91d342f1af627e19f3e04f53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:59.173475 29523 raft_consensus.cc:729] T 2a6efa0635c243e689b704c218d012f0 P 053fb01d91d342f1af627e19f3e04f53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 053fb01d91d342f1af627e19f3e04f53, State: Initialized, Role: FOLLOWER
I20260619 18:42:59.173604 29523 consensus_queue.cc:260] T 2a6efa0635c243e689b704c218d012f0 P 053fb01d91d342f1af627e19f3e04f53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0928d174307d4be2bf5124d798582bf0" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33201 } } peers { permanent_uuid: "580c2c8aeac04a6899c26151d90802ca" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 38767 } } peers { permanent_uuid: "053fb01d91d342f1af627e19f3e04f53" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44621 } }
I20260619 18:42:59.174003 29150 heartbeater.cc:503] Master 127.21.143.124:37717 was elected leader, sending a full tablet report...
I20260619 18:42:59.174000 29523 ts_tablet_manager.cc:1428] T 2a6efa0635c243e689b704c218d012f0 P 053fb01d91d342f1af627e19f3e04f53: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:42:59.175709 29509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:59.175482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.21.143.65:59729)
0619 18:42:59.175589 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:59.175645 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 18:42:59.176605 29509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:59.176144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.21.143.65:42683)
0619 18:42:59.176481 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:59.176541 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271}
I20260619 18:42:59.178331 29528 raft_consensus.cc:482] T 2a6efa0635c243e689b704c218d012f0 P 580c2c8aeac04a6899c26151d90802ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:42:59.178427 29528 raft_consensus.cc:504] T 2a6efa0635c243e689b704c218d012f0 P 580c2c8aeac04a6899c26151d90802ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0928d174307d4be2bf5124d798582bf0" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33201 } } peers { permanent_uuid: "580c2c8aeac04a6899c26151d90802ca" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 38767 } } peers { permanent_uuid: "053fb01d91d342f1af627e19f3e04f53" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44621 } }
I20260619 18:42:59.179571 29528 leader_election.cc:288] T 2a6efa0635c243e689b704c218d012f0 P 580c2c8aeac04a6899c26151d90802ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0928d174307d4be2bf5124d798582bf0 (127.21.143.67:33201), 053fb01d91d342f1af627e19f3e04f53 (127.21.143.65:44621)
I20260619 18:42:59.183131 29516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:59.179964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:51651 (local address 127.21.143.67:33201)
0619 18:42:59.180103 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:59.180107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:59.180118 (+ 11us) server_negotiation.cc:407] Connection header received
0619 18:42:59.180155 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:59.180157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:59.180190 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:59.180281 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:59.180537 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:59.181060 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:59.182159 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:59.182570 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:59.182696 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:59.182748 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:59.182927 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 18:42:59.182998 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 18:42:59.183173 29508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:59.179728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:33201 (local address 127.21.143.66:51651)
0619 18:42:59.179887 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:59.179904 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:59.180000 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:59.180292 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:59.180295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:59.180313 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:59.180432 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:59.180437 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:59.181167 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:59.181173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:59.182035 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:59.182044 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:59.182692 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:59.182695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:59.182797 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:59.182825 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 18:42:59.182914 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 18:42:59.182989 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":20}
I20260619 18:42:59.183194 29520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:59.179973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:40291 (local address 127.21.143.65:44621)
0619 18:42:59.180107 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:59.180111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:59.180261 (+ 150us) server_negotiation.cc:407] Connection header received
0619 18:42:59.180370 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:59.180373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:59.180410 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:59.180493 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:59.180726 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:59.181217 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:59.182221 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:59.182655 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:59.182777 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:59.182826 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:59.182991 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 18:42:59.183075 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 18:42:59.183212 29530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:59.179726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:44621 (local address 127.21.143.66:40291)
0619 18:42:59.180178 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:59.180192 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:59.180294 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:59.180519 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:59.180522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:59.180537 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:59.180621 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:59.180633 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:59.181325 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:59.181329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:59.182097 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:59.182106 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:59.182777 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:59.182781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:59.182866 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:59.182886 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 18:42:59.182977 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 18:42:59.183047 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":71,"thread_start_us":103,"threads_started":1}
I20260619 18:42:59.183712 29377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6efa0635c243e689b704c218d012f0" candidate_uuid: "580c2c8aeac04a6899c26151d90802ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0928d174307d4be2bf5124d798582bf0" is_pre_election: true
I20260619 18:42:59.183748 29104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6efa0635c243e689b704c218d012f0" candidate_uuid: "580c2c8aeac04a6899c26151d90802ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "053fb01d91d342f1af627e19f3e04f53" is_pre_election: true
I20260619 18:42:59.183862 29377 raft_consensus.cc:2454] T 2a6efa0635c243e689b704c218d012f0 P 0928d174307d4be2bf5124d798582bf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 580c2c8aeac04a6899c26151d90802ca in term 0.
I20260619 18:42:59.183979 29104 raft_consensus.cc:2454] T 2a6efa0635c243e689b704c218d012f0 P 053fb01d91d342f1af627e19f3e04f53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 580c2c8aeac04a6899c26151d90802ca in term 0.
I20260619 18:42:59.184264 29176 leader_election.cc:302] T 2a6efa0635c243e689b704c218d012f0 P 580c2c8aeac04a6899c26151d90802ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0928d174307d4be2bf5124d798582bf0, 580c2c8aeac04a6899c26151d90802ca; no voters:
I20260619 18:42:59.184573 29528 raft_consensus.cc:2781] T 2a6efa0635c243e689b704c218d012f0 P 580c2c8aeac04a6899c26151d90802ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:42:59.184638 29528 raft_consensus.cc:482] T 2a6efa0635c243e689b704c218d012f0 P 580c2c8aeac04a6899c26151d90802ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:42:59.184685 29528 raft_consensus.cc:3037] T 2a6efa0635c243e689b704c218d012f0 P 580c2c8aeac04a6899c26151d90802ca [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:59.185668 29528 raft_consensus.cc:504] T 2a6efa0635c243e689b704c218d012f0 P 580c2c8aeac04a6899c26151d90802ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0928d174307d4be2bf5124d798582bf0" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33201 } } peers { permanent_uuid: "580c2c8aeac04a6899c26151d90802ca" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 38767 } } peers { permanent_uuid: "053fb01d91d342f1af627e19f3e04f53" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44621 } }
I20260619 18:42:59.185999 29528 leader_election.cc:288] T 2a6efa0635c243e689b704c218d012f0 P 580c2c8aeac04a6899c26151d90802ca [CANDIDATE]: Term 1 election: Requested vote from peers 0928d174307d4be2bf5124d798582bf0 (127.21.143.67:33201), 053fb01d91d342f1af627e19f3e04f53 (127.21.143.65:44621)
I20260619 18:42:59.186357 29377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6efa0635c243e689b704c218d012f0" candidate_uuid: "580c2c8aeac04a6899c26151d90802ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0928d174307d4be2bf5124d798582bf0"
I20260619 18:42:59.186362 29104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6efa0635c243e689b704c218d012f0" candidate_uuid: "580c2c8aeac04a6899c26151d90802ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "053fb01d91d342f1af627e19f3e04f53"
I20260619 18:42:59.186461 29104 raft_consensus.cc:3037] T 2a6efa0635c243e689b704c218d012f0 P 053fb01d91d342f1af627e19f3e04f53 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:59.186488 29377 raft_consensus.cc:3037] T 2a6efa0635c243e689b704c218d012f0 P 0928d174307d4be2bf5124d798582bf0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:59.187415 29377 raft_consensus.cc:2454] T 2a6efa0635c243e689b704c218d012f0 P 0928d174307d4be2bf5124d798582bf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 580c2c8aeac04a6899c26151d90802ca in term 1.
I20260619 18:42:59.187500 29104 raft_consensus.cc:2454] T 2a6efa0635c243e689b704c218d012f0 P 053fb01d91d342f1af627e19f3e04f53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 580c2c8aeac04a6899c26151d90802ca in term 1.
I20260619 18:42:59.187776 29176 leader_election.cc:302] T 2a6efa0635c243e689b704c218d012f0 P 580c2c8aeac04a6899c26151d90802ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0928d174307d4be2bf5124d798582bf0, 580c2c8aeac04a6899c26151d90802ca; no voters:
I20260619 18:42:59.187958 29528 raft_consensus.cc:2781] T 2a6efa0635c243e689b704c218d012f0 P 580c2c8aeac04a6899c26151d90802ca [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:59.188185 29528 raft_consensus.cc:686] T 2a6efa0635c243e689b704c218d012f0 P 580c2c8aeac04a6899c26151d90802ca [term 1 LEADER]: Becoming Leader. State: Replica: 580c2c8aeac04a6899c26151d90802ca, State: Running, Role: LEADER
I20260619 18:42:59.188453 29528 consensus_queue.cc:237] T 2a6efa0635c243e689b704c218d012f0 P 580c2c8aeac04a6899c26151d90802ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0928d174307d4be2bf5124d798582bf0" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33201 } } peers { permanent_uuid: "580c2c8aeac04a6899c26151d90802ca" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 38767 } } peers { permanent_uuid: "053fb01d91d342f1af627e19f3e04f53" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44621 } }
I20260619 18:42:59.188983 29530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:59.188668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.21.143.66:46159)
0619 18:42:59.188854 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:59.188919 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260619 18:42:59.191105 28971 catalog_manager.cc:5495] T 2a6efa0635c243e689b704c218d012f0 P 580c2c8aeac04a6899c26151d90802ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 580c2c8aeac04a6899c26151d90802ca (127.21.143.66). New cstate: current_term: 1 leader_uuid: "580c2c8aeac04a6899c26151d90802ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0928d174307d4be2bf5124d798582bf0" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "580c2c8aeac04a6899c26151d90802ca" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 38767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "053fb01d91d342f1af627e19f3e04f53" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44621 } health_report { overall_health: UNKNOWN } } }
I20260619 18:42:59.211933 29519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:59.208442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33588 (local address 127.21.143.66:38767)
0619 18:42:59.208566 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:59.208570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:59.208583 (+ 13us) server_negotiation.cc:407] Connection header received
0619 18:42:59.208659 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:59.208661 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:59.208715 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:59.208814 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:42:59.209107 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:59.209636 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:59.210460 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:59.210674 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:59.210758 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:59.210993 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:42:59.211533 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:42:59.211654 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:59.211717 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 18:42:59.211768 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 18:42:59.214727 29377 raft_consensus.cc:1261] T 2a6efa0635c243e689b704c218d012f0 P 0928d174307d4be2bf5124d798582bf0 [term 1 FOLLOWER]: Refusing update from remote peer 580c2c8aeac04a6899c26151d90802ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:42:59.214977 29104 raft_consensus.cc:1261] T 2a6efa0635c243e689b704c218d012f0 P 053fb01d91d342f1af627e19f3e04f53 [term 1 FOLLOWER]: Refusing update from remote peer 580c2c8aeac04a6899c26151d90802ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:42:59.215245 29528 consensus_queue.cc:1035] T 2a6efa0635c243e689b704c218d012f0 P 580c2c8aeac04a6899c26151d90802ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "0928d174307d4be2bf5124d798582bf0" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:42:59.215590 29531 consensus_queue.cc:1035] T 2a6efa0635c243e689b704c218d012f0 P 580c2c8aeac04a6899c26151d90802ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "053fb01d91d342f1af627e19f3e04f53" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:42:59.217164 29507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:59.216886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.21.143.67:36113)
0619 18:42:59.217034 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:59.217091 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 18:42:59.218740 29509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:59.218191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.21.143.65:51331)
0619 18:42:59.218611 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:59.218669 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347}
I20260619 18:42:59.220893 29535 mvcc.cc:205] Tried to move back new op lower bound from 7298640196458520576 to 7298640196360011776. Current Snapshot: MvccSnapshot[applied={T|T < 7298640196458520576}]
I20260619 18:42:59.221565 29507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:59.221300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.21.143.67:36615)
0619 18:42:59.221433 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:59.221488 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 18:42:59.221743 29530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:59.221529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.21.143.66:48927)
0619 18:42:59.221621 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:59.221673 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 18:42:59.221777 29509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:59.221221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.21.143.65:56239)
0619 18:42:59.221676 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:59.221717 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393}
I20260619 18:42:59.222577 29530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:59.222321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:39877 (local address 127.21.143.66:43843)
0619 18:42:59.222460 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:59.222508 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 18:42:59.230425 28971 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:42:59.230571 28971 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:42:59.234391 28971 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dab52bca615747c89249a864e1553522: Sending DeleteTablet for 3 replicas of tablet 2a6efa0635c243e689b704c218d012f0
I20260619 18:42:59.235131 29084 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a6efa0635c243e689b704c218d012f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:42:59 UTC) from {username='oryx', principal='oryx/127.21.143.124@KRBTEST.COM'} at 127.0.0.1:41804
I20260619 18:42:59.235123 29221 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a6efa0635c243e689b704c218d012f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:42:59 UTC) from {username='oryx', principal='oryx/127.21.143.124@KRBTEST.COM'} at 127.0.0.1:33584
I20260619 18:42:59.235263 29357 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a6efa0635c243e689b704c218d012f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:42:59 UTC) from {username='oryx', principal='oryx/127.21.143.124@KRBTEST.COM'} at 127.0.0.1:51366
I20260619 18:42:59.235684 29545 tablet_replica.cc:330] T 2a6efa0635c243e689b704c218d012f0 P 580c2c8aeac04a6899c26151d90802ca: stopping tablet replica
I20260619 18:42:59.235800 29545 raft_consensus.cc:2229] T 2a6efa0635c243e689b704c218d012f0 P 580c2c8aeac04a6899c26151d90802ca [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:42:59.235968 29546 tablet_replica.cc:330] T 2a6efa0635c243e689b704c218d012f0 P 0928d174307d4be2bf5124d798582bf0: stopping tablet replica
I20260619 18:42:59.236119 29545 raft_consensus.cc:2258] T 2a6efa0635c243e689b704c218d012f0 P 580c2c8aeac04a6899c26151d90802ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:42:59.236148 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 28932
I20260619 18:42:59.236146 29546 raft_consensus.cc:2229] T 2a6efa0635c243e689b704c218d012f0 P 0928d174307d4be2bf5124d798582bf0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:42:59.236872 29546 raft_consensus.cc:2258] T 2a6efa0635c243e689b704c218d012f0 P 0928d174307d4be2bf5124d798582bf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:42:59.244762 29545 ts_tablet_manager.cc:1905] T 2a6efa0635c243e689b704c218d012f0 P 580c2c8aeac04a6899c26151d90802ca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 18:42:59.245532 29517 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:59.244532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:44621 (local address 127.0.0.1:41818)
0619 18:42:59.244702 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:59.244717 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:59.244804 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:59.245422 (+ 618us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.65:44621: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":94}
W20260619 18:42:59.245733 28956 catalog_manager.cc:4584] TS 053fb01d91d342f1af627e19f3e04f53 (127.21.143.65:44621): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a6efa0635c243e689b704c218d012f0: Network error: Client connection negotiation failed: client connection to 127.21.143.65:44621: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 18:42:59.246280 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 29158
I20260619 18:42:59.252074 29546 ts_tablet_manager.cc:1905] T 2a6efa0635c243e689b704c218d012f0 P 0928d174307d4be2bf5124d798582bf0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:42:59.253489 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 29294
I20260619 18:42:59.253602 29517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:59.253326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:38767 (local address 127.0.0.1:33592)
0619 18:42:59.253444 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:59.253504 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.66:38767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 18:42:59.259446 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 28649
I20260619 18:42:59.260010 29517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:59.259536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:33201 (local address 127.0.0.1:51368)
0619 18:42:59.259898 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:59.259942 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.67:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286}
I20260619 18:42:59.265540 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 28753
2026-06-19T18:42:59Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5825 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:42:59 dist-test-slave-mld5 krb5kdc[29549](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:42:59 dist-test-slave-mld5 krb5kdc[29549](info): set up 2 sockets
Jun 19 18:42:59 dist-test-slave-mld5 krb5kdc[29549](info): commencing operation
krb5kdc: starting...
W20260619 18:43:00.295989 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:43:00 dist-test-slave-mld5 krb5kdc[29549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:43:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:43:00Z Disabled control of system clock
WARNING: no policy specified for oryx/127.21.143.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.143.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.143.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.126.keytab.
Entry for principal oryx/127.21.143.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.126.keytab.
I20260619 18:43:00.414462 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:33147
--webserver_interface=127.21.143.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.126.keytab
--principal=oryx/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:37981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:33147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:43:00.523461 29562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:43:00.523787 29562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:43:00.523869 29562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:43:00.527985 29562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:43:00.528064 29562 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 18:43:00.528121 29562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:43:00.528182 29562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:43:00.528223 29562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:43:00.528259 29562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:43:00.533679 29562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:37981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:33147
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.126.keytab
--principal=oryx/127.21.143.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:33147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:43:00.534886 29562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:43:00.535969 29562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:43:00.544062 29568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:00.544063 29570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:00.544085 29567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:00.544415 29562 server_base.cc:1034] running on GCE node
I20260619 18:43:00.544997 29562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:43:00.546056 29562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:43:00.547235 29562 hybrid_clock.cc:648] HybridClock initialized: now 1781894580547205 us; error 49 us; skew 500 ppm
Jun 19 18:43:00 dist-test-slave-mld5 krb5kdc[29549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894580, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:43:00.550045 29562 init.cc:377] Logged in from keytab as oryx/127.21.143.126@KRBTEST.COM (short username oryx)
I20260619 18:43:00.550891 29562 webserver.cc:460] Webserver started at http://127.21.143.126:34523/ using document root <none> and password file <none>
I20260619 18:43:00.551503 29562 fs_manager.cc:359] Metadata directory not provided
I20260619 18:43:00.551592 29562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:43:00.551822 29562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:43:00.553773 29562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "a3f24326df2f48d7bff275e80e8cbcfe"
format_stamp: "Formatted at 2026-06-19 18:43:00 on dist-test-slave-mld5"
server_key: "e5f11eb0c49cea2c499149c24cb842f7"
server_key_iv: "8038bff2ea0626291a8883a88620709f"
server_key_version: "encryptionkey@0"
I20260619 18:43:00.554330 29562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "a3f24326df2f48d7bff275e80e8cbcfe"
format_stamp: "Formatted at 2026-06-19 18:43:00 on dist-test-slave-mld5"
server_key: "e5f11eb0c49cea2c499149c24cb842f7"
server_key_iv: "8038bff2ea0626291a8883a88620709f"
server_key_version: "encryptionkey@0"
I20260619 18:43:00.557921 29562 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 18:43:00.563581 29577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:00.565147 29562 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260619 18:43:00.565267 29562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "a3f24326df2f48d7bff275e80e8cbcfe"
format_stamp: "Formatted at 2026-06-19 18:43:00 on dist-test-slave-mld5"
server_key: "e5f11eb0c49cea2c499149c24cb842f7"
server_key_iv: "8038bff2ea0626291a8883a88620709f"
server_key_version: "encryptionkey@0"
I20260619 18:43:00.565402 29562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:43:00.590022 29562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:43:00.593083 29562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:43:00.593290 29562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:43:00.601163 29562 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:33147
I20260619 18:43:00.601193 29639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:33147 every 8 connection(s)
I20260619 18:43:00.602094 29562 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:43:00.604779 29640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:43:00.610033 29640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3f24326df2f48d7bff275e80e8cbcfe: Bootstrap starting.
I20260619 18:43:00.610837 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 29562
I20260619 18:43:00.610967 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:43:00.611291 22077 external_mini_cluster.cc:1428] Setting key cfdb349aeeb6c00663bb63e8669268dd
I20260619 18:43:00.612874 29640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3f24326df2f48d7bff275e80e8cbcfe: Neither blocks nor log segments found. Creating new log.
I20260619 18:43:00.613570 29640 log.cc:824] T 00000000000000000000000000000000 P a3f24326df2f48d7bff275e80e8cbcfe: Log is configured to *not* fsync() on all Append() calls
I20260619 18:43:00.615690 29640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3f24326df2f48d7bff275e80e8cbcfe: No bootstrap required, opened a new log
Jun 19 18:43:00 dist-test-slave-mld5 krb5kdc[29549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM
I20260619 18:43:00.620081 29640 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3f24326df2f48d7bff275e80e8cbcfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f24326df2f48d7bff275e80e8cbcfe" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33147 } }
I20260619 18:43:00.620312 29640 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3f24326df2f48d7bff275e80e8cbcfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:43:00.620369 29640 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3f24326df2f48d7bff275e80e8cbcfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3f24326df2f48d7bff275e80e8cbcfe, State: Initialized, Role: FOLLOWER
I20260619 18:43:00.620877 29640 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3f24326df2f48d7bff275e80e8cbcfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f24326df2f48d7bff275e80e8cbcfe" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33147 } }
I20260619 18:43:00.621043 29640 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3f24326df2f48d7bff275e80e8cbcfe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:43:00.621129 29640 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3f24326df2f48d7bff275e80e8cbcfe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:43:00.621219 29640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3f24326df2f48d7bff275e80e8cbcfe [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:43:00.622350 29640 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3f24326df2f48d7bff275e80e8cbcfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f24326df2f48d7bff275e80e8cbcfe" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33147 } }
I20260619 18:43:00.622710 29640 leader_election.cc:302] T 00000000000000000000000000000000 P a3f24326df2f48d7bff275e80e8cbcfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3f24326df2f48d7bff275e80e8cbcfe; no voters:
I20260619 18:43:00.623047 29640 leader_election.cc:288] T 00000000000000000000000000000000 P a3f24326df2f48d7bff275e80e8cbcfe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:43:00.623106 29645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3f24326df2f48d7bff275e80e8cbcfe [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:43:00.623328 29645 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3f24326df2f48d7bff275e80e8cbcfe [term 1 LEADER]: Becoming Leader. State: Replica: a3f24326df2f48d7bff275e80e8cbcfe, State: Running, Role: LEADER
I20260619 18:43:00.623592 29645 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3f24326df2f48d7bff275e80e8cbcfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f24326df2f48d7bff275e80e8cbcfe" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33147 } }
I20260619 18:43:00.624279 29640 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3f24326df2f48d7bff275e80e8cbcfe [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:43:00.625268 29646 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3f24326df2f48d7bff275e80e8cbcfe [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3f24326df2f48d7bff275e80e8cbcfe. Latest consensus state: current_term: 1 leader_uuid: "a3f24326df2f48d7bff275e80e8cbcfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f24326df2f48d7bff275e80e8cbcfe" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33147 } } }
I20260619 18:43:00.625402 29646 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3f24326df2f48d7bff275e80e8cbcfe [sys.catalog]: This master's current role is: LEADER
I20260619 18:43:00.625339 29647 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3f24326df2f48d7bff275e80e8cbcfe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3f24326df2f48d7bff275e80e8cbcfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f24326df2f48d7bff275e80e8cbcfe" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33147 } } }
I20260619 18:43:00.625603 29647 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3f24326df2f48d7bff275e80e8cbcfe [sys.catalog]: This master's current role is: LEADER
I20260619 18:43:00.625877 29654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:43:00.626174 29643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:00.612406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46482 (local address 127.21.143.126:33147)
0619 18:43:00.612791 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:00.612806 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:00.612841 (+ 35us) server_negotiation.cc:407] Connection header received
0619 18:43:00.613502 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:00.613519 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:00.613913 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:00.614280 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:00.615117 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:00.616210 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:00.617430 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:00.617905 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:00.618020 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:00.620820 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:00.620826 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:00.620841 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:00.620869 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:00.622762 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:00.623200 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:00.623208 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:00.623218 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:00.623382 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:00.623697 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:00.623703 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:00.623708 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:00.624197 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:00.624404 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:00.624704 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 18:43:00.625018 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":135,"threads_started":1}
I20260619 18:43:00.629539 29654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:43:00.634634 29654 catalog_manager.cc:1347] Generated new cluster ID: f45ccb0b243b444fbfdc9b247eee96ba
I20260619 18:43:00.634697 29654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:43:00.644152 29654 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:43:00.645098 29654 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:43:00.662650 29654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3f24326df2f48d7bff275e80e8cbcfe: Generated new TSK 0
I20260619 18:43:00.663465 29654 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.21.143.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.143.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.143.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.65.keytab.
Entry for principal oryx/127.21.143.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.65.keytab.
I20260619 18:43:00.692080 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.65:0
--local_ip_for_outbound_sockets=127.21.143.65
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:33147
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.65.keytab
--principal=oryx/127.21.143.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:37981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:43:00.833612 29666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:43:00.833904 29666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:43:00.833946 29666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:43:00.837791 29666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:43:00.837841 29666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:43:00.837877 29666 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:43:00.837909 29666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.65
I20260619 18:43:00.842857 29666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:37981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.65.keytab
--principal=oryx/127.21.143.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.143.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.143.126:33147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.65
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:43:00.843820 29666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:43:00.844761 29666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:43:00.853763 29674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:00.853796 29672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:00.853772 29671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:00.853963 29666 server_base.cc:1034] running on GCE node
I20260619 18:43:00.854420 29666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:43:00.855019 29666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:43:00.856173 29666 hybrid_clock.cc:648] HybridClock initialized: now 1781894580856134 us; error 49 us; skew 500 ppm
Jun 19 18:43:00 dist-test-slave-mld5 krb5kdc[29549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894580, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:43:00.858844 29666 init.cc:377] Logged in from keytab as oryx/127.21.143.65@KRBTEST.COM (short username oryx)
I20260619 18:43:00.859874 29666 webserver.cc:460] Webserver started at http://127.21.143.65:41779/ using document root <none> and password file <none>
I20260619 18:43:00.860484 29666 fs_manager.cc:359] Metadata directory not provided
I20260619 18:43:00.860549 29666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:43:00.860821 29666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:43:00.863006 29666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-0/data/instance:
uuid: "045f04c1759a4d9bbf38cc06b397ba76"
format_stamp: "Formatted at 2026-06-19 18:43:00 on dist-test-slave-mld5"
server_key: "ba7d84038b7aa09876fd0b5d1ba930f9"
server_key_iv: "128b64417a5abec9c51ddea9e8e29809"
server_key_version: "encryptionkey@0"
I20260619 18:43:00.863668 29666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance:
uuid: "045f04c1759a4d9bbf38cc06b397ba76"
format_stamp: "Formatted at 2026-06-19 18:43:00 on dist-test-slave-mld5"
server_key: "ba7d84038b7aa09876fd0b5d1ba930f9"
server_key_iv: "128b64417a5abec9c51ddea9e8e29809"
server_key_version: "encryptionkey@0"
I20260619 18:43:00.867702 29666 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:43:00.870390 29681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:00.871837 29666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:43:00.871950 29666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-0/wal
uuid: "045f04c1759a4d9bbf38cc06b397ba76"
format_stamp: "Formatted at 2026-06-19 18:43:00 on dist-test-slave-mld5"
server_key: "ba7d84038b7aa09876fd0b5d1ba930f9"
server_key_iv: "128b64417a5abec9c51ddea9e8e29809"
server_key_version: "encryptionkey@0"
I20260619 18:43:00.872112 29666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:43:00.884778 29666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:43:00.888093 29666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:43:00.888335 29666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:43:00.889889 29666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:43:00.889940 29666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:00.889981 29666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:43:00.890048 29666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:43:00 dist-test-slave-mld5 krb5kdc[29549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894580, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM
I20260619 18:43:00.901307 29666 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.65:36541
I20260619 18:43:00.901331 29796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.65:36541 every 8 connection(s)
I20260619 18:43:00.902513 29666 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
I20260619 18:43:00.905032 29643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:00.890847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:59967 (local address 127.21.143.126:33147)
0619 18:43:00.890979 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:00.890982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:00.892148 (+ 1166us) server_negotiation.cc:407] Connection header received
0619 18:43:00.893451 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:00.893458 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:00.893533 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:00.893641 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:00.895342 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:00.896261 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:00.897828 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:00.898198 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:00.898316 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:00.901082 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:00.901089 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:00.901093 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:00.901122 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:00.902857 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:00.903581 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:00.903585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:00.903587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:00.903712 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:00.904037 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:00.904039 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:00.904040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:00.904238 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:00.904315 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:00.904801 (+ 486us) server_negotiation.cc:299] Negotiation successful
0619 18:43:00.904910 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 18:43:00.905900 29690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:00.891474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33147 (local address 127.21.143.65:59967)
0619 18:43:00.891977 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:00.892021 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:00.893172 (+ 1151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:00.893873 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:00.893880 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:00.894360 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:00.895115 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:00.895147 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:00.896407 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:00.896411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:00.897718 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:00.897728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:00.898352 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:00.898355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:00.898493 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:00.899199 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:00.899219 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:00.900890 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:43:00.903006 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:00.903012 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:00.903027 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:00.903435 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:00.903809 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:00.903811 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:00.903813 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:00.903948 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:00.904336 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:43:00.904341 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:43:00.904584 (+ 243us) client_negotiation.cc:769] Sending connection context
0619 18:43:00.904741 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 18:43:00.904999 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":155,"threads_started":1}
I20260619 18:43:00.907188 29797 heartbeater.cc:347] Connected to a master server at 127.21.143.126:33147
I20260619 18:43:00.907353 29797 heartbeater.cc:464] Registering TS with master...
I20260619 18:43:00.907941 29797 heartbeater.cc:511] Master 127.21.143.126:33147 requested a full tablet report, sending...
I20260619 18:43:00.908371 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 29666
I20260619 18:43:00.908505 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance
I20260619 18:43:00.908850 22077 external_mini_cluster.cc:1428] Setting key 9057ae29a1508ab25cd7217731831ad3
I20260619 18:43:00.910627 29594 ts_manager.cc:194] Registered new tserver with Master: 045f04c1759a4d9bbf38cc06b397ba76 (127.21.143.65:36541)
I20260619 18:43:00.912261 29594 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.143.65@KRBTEST.COM'} at 127.21.143.65:59967
I20260619 18:43:00.918334 29643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:00.909725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:45255 (local address 127.21.143.126:33147)
0619 18:43:00.910334 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:00.910338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:00.910351 (+ 13us) server_negotiation.cc:407] Connection header received
0619 18:43:00.910385 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:00.910387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:00.910431 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:00.910514 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:00.910998 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:00.911594 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:00.913761 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:00.913991 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:00.914099 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:00.915253 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:00.915257 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:00.915260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:00.915288 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:00.916724 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:00.917145 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:00.917149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:00.917151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:00.917232 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:00.917512 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:00.917517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:00.917518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:00.917701 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:00.917809 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:00.918067 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 18:43:00.918177 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541}
I20260619 18:43:00.918351 29803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:00.909677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33147 (local address 127.21.143.65:45255)
0619 18:43:00.909965 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:00.909980 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:00.910063 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:00.910532 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:00.910535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:00.910772 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:00.910887 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:00.910893 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:00.911717 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:00.911721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:00.913641 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:00.913651 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:00.914104 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:00.914107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:00.914204 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:00.914762 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:00.914773 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:00.915087 (+ 314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:43:00.916843 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:00.916846 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:00.916848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:00.917046 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:00.917331 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:00.917334 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:00.917336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:00.917423 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:00.917799 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:43:00.917802 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:43:00.917895 (+ 93us) client_negotiation.cc:769] Sending connection context
0619 18:43:00.918005 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 18:43:00.918149 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for oryx/127.21.143.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.143.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.143.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.66.keytab.
Entry for principal oryx/127.21.143.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.66.keytab.
I20260619 18:43:00.936125 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.66:0
--local_ip_for_outbound_sockets=127.21.143.66
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:33147
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.66.keytab
--principal=oryx/127.21.143.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:37981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:43:01.045087 29806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:43:01.045403 29806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:43:01.045449 29806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:43:01.049715 29806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:43:01.049798 29806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:43:01.049841 29806 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:43:01.049914 29806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.66
I20260619 18:43:01.055680 29806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:37981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.66.keytab
--principal=oryx/127.21.143.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.143.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.143.126:33147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.66
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:43:01.056870 29806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:43:01.057974 29806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:43:01.067899 29814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:01.067911 29811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:01.067904 29812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:01.068284 29806 server_base.cc:1034] running on GCE node
I20260619 18:43:01.068672 29806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:43:01.069267 29806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:43:01.070459 29806 hybrid_clock.cc:648] HybridClock initialized: now 1781894581070442 us; error 30 us; skew 500 ppm
Jun 19 18:43:01 dist-test-slave-mld5 krb5kdc[29549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894581, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:43:01.073323 29806 init.cc:377] Logged in from keytab as oryx/127.21.143.66@KRBTEST.COM (short username oryx)
I20260619 18:43:01.074167 29806 webserver.cc:460] Webserver started at http://127.21.143.66:33421/ using document root <none> and password file <none>
I20260619 18:43:01.074736 29806 fs_manager.cc:359] Metadata directory not provided
I20260619 18:43:01.074824 29806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:43:01.075048 29806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:43:01.077173 29806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-1/data/instance:
uuid: "b9d3e9ca17bd4d9babad18338ae019ee"
format_stamp: "Formatted at 2026-06-19 18:43:01 on dist-test-slave-mld5"
server_key: "90705c519d6eeca54cbe573e0eadd7a9"
server_key_iv: "89cae8b457f824f446fc126a1149210b"
server_key_version: "encryptionkey@0"
I20260619 18:43:01.077718 29806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance:
uuid: "b9d3e9ca17bd4d9babad18338ae019ee"
format_stamp: "Formatted at 2026-06-19 18:43:01 on dist-test-slave-mld5"
server_key: "90705c519d6eeca54cbe573e0eadd7a9"
server_key_iv: "89cae8b457f824f446fc126a1149210b"
server_key_version: "encryptionkey@0"
I20260619 18:43:01.081339 29806 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:43:01.083706 29821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:01.084920 29806 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:43:01.085057 29806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-1/wal
uuid: "b9d3e9ca17bd4d9babad18338ae019ee"
format_stamp: "Formatted at 2026-06-19 18:43:01 on dist-test-slave-mld5"
server_key: "90705c519d6eeca54cbe573e0eadd7a9"
server_key_iv: "89cae8b457f824f446fc126a1149210b"
server_key_version: "encryptionkey@0"
I20260619 18:43:01.085165 29806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:43:01.095990 29806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:43:01.098496 29806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:43:01.098690 29806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:43:01.100066 29806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:43:01.100136 29806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:01.100205 29806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:43:01.100251 29806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:43:01 dist-test-slave-mld5 krb5kdc[29549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894581, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM
I20260619 18:43:01.110942 29806 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.66:39881
I20260619 18:43:01.111016 29936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.66:39881 every 8 connection(s)
I20260619 18:43:01.112228 29806 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
I20260619 18:43:01.114703 29643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.100835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:55107 (local address 127.21.143.126:33147)
0619 18:43:01.101001 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:01.101007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:01.101853 (+ 846us) server_negotiation.cc:407] Connection header received
0619 18:43:01.102853 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:01.102857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:01.102916 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:01.102980 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:01.104763 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.105466 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.107191 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.107668 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.107803 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.110780 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:01.110785 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:01.110788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:01.110823 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:01.112613 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:01.113226 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:01.113229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:01.113231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:01.113328 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:01.113703 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:01.113705 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:01.113706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:01.113900 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:01.113975 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:01.114457 (+ 482us) server_negotiation.cc:299] Negotiation successful
0619 18:43:01.114585 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 18:43:01.115567 29830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.101212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33147 (local address 127.21.143.66:55107)
0619 18:43:01.101689 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:01.101736 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:01.102600 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:01.103442 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:01.103451 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:01.103903 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:01.104600 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:01.104625 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.105589 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.105593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:01.107063 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:01.107075 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.107888 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.107894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:01.108116 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.108795 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:01.108811 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:01.110569 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:43:01.112758 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:01.112763 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:01.112778 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:01.113122 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:01.113441 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:01.113444 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:01.113447 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:01.113618 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:01.113987 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:43:01.113992 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:43:01.114220 (+ 228us) client_negotiation.cc:769] Sending connection context
0619 18:43:01.114394 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 18:43:01.114646 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":134,"threads_started":1}
I20260619 18:43:01.116765 29937 heartbeater.cc:347] Connected to a master server at 127.21.143.126:33147
I20260619 18:43:01.116909 29937 heartbeater.cc:464] Registering TS with master...
I20260619 18:43:01.117451 29937 heartbeater.cc:511] Master 127.21.143.126:33147 requested a full tablet report, sending...
I20260619 18:43:01.118510 29594 ts_manager.cc:194] Registered new tserver with Master: b9d3e9ca17bd4d9babad18338ae019ee (127.21.143.66:39881)
I20260619 18:43:01.119334 29594 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.143.66@KRBTEST.COM'} at 127.21.143.66:55107
I20260619 18:43:01.121757 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 29806
I20260619 18:43:01.121903 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance
I20260619 18:43:01.122193 22077 external_mini_cluster.cc:1428] Setting key ba5a767bb744c68f66947d142487fd83
I20260619 18:43:01.126443 29943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.118986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33147 (local address 127.21.143.66:60691)
0619 18:43:01.119240 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:01.119252 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:01.119398 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:01.119657 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:01.119661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:01.119844 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:01.119965 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:01.119971 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.120702 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.120705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:01.121260 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:01.121267 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.121691 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.121694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:01.121789 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.122309 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:01.122328 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:01.122743 (+ 415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:43:01.124777 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:01.124779 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:01.124781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:01.125022 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:01.125362 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:01.125368 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:01.125373 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:01.125469 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:01.126015 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:43:01.126017 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:43:01.126094 (+ 77us) client_negotiation.cc:769] Sending connection context
0619 18:43:01.126177 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 18:43:01.126284 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":21,"thread_start_us":96,"threads_started":1}
I20260619 18:43:01.126614 29643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.119046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:60691 (local address 127.21.143.126:33147)
0619 18:43:01.119160 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:01.119164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:01.119336 (+ 172us) server_negotiation.cc:407] Connection header received
0619 18:43:01.119494 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:01.119497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:01.119540 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:01.119643 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:01.120072 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.120569 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.121374 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.121579 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.121677 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.122866 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:01.122870 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:01.122873 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:01.122907 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:01.124677 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:01.125147 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:01.125154 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:01.125158 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:01.125248 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:01.125599 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:01.125605 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:01.125609 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:01.125888 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:01.125999 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:01.126287 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 18:43:01.126420 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
WARNING: no policy specified for oryx/127.21.143.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.143.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.143.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.67.keytab.
Entry for principal oryx/127.21.143.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.67.keytab.
I20260619 18:43:01.148236 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.67:0
--local_ip_for_outbound_sockets=127.21.143.67
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:33147
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.67.keytab
--principal=oryx/127.21.143.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:37981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:43:01.263149 29946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:43:01.263556 29946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:43:01.263687 29946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:43:01.268831 29946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:43:01.268909 29946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:43:01.268951 29946 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:43:01.268987 29946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.67
I20260619 18:43:01.274688 29946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:37981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.67.keytab
--principal=oryx/127.21.143.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.143.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.143.126:33147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.67
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:43:01.275911 29946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:43:01.277048 29946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:43:01.286810 29952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:01.286857 29954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:01.286810 29951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:01.287003 29946 server_base.cc:1034] running on GCE node
I20260619 18:43:01.287623 29946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:43:01.288291 29946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:43:01.289482 29946 hybrid_clock.cc:648] HybridClock initialized: now 1781894581289462 us; error 33 us; skew 500 ppm
Jun 19 18:43:01 dist-test-slave-mld5 krb5kdc[29549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894581, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:43:01.292339 29946 init.cc:377] Logged in from keytab as oryx/127.21.143.67@KRBTEST.COM (short username oryx)
I20260619 18:43:01.293205 29946 webserver.cc:460] Webserver started at http://127.21.143.67:39973/ using document root <none> and password file <none>
I20260619 18:43:01.293776 29946 fs_manager.cc:359] Metadata directory not provided
I20260619 18:43:01.293836 29946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:43:01.294000 29946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:43:01.296015 29946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-2/data/instance:
uuid: "5ee365d535f64fda9b04ac44dd08738c"
format_stamp: "Formatted at 2026-06-19 18:43:01 on dist-test-slave-mld5"
server_key: "eae4db4b7dbbd3f6ad3bef46ae4feb24"
server_key_iv: "56ddea69aae055be916bb59ac7d7ecb8"
server_key_version: "encryptionkey@0"
I20260619 18:43:01.296519 29946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance:
uuid: "5ee365d535f64fda9b04ac44dd08738c"
format_stamp: "Formatted at 2026-06-19 18:43:01 on dist-test-slave-mld5"
server_key: "eae4db4b7dbbd3f6ad3bef46ae4feb24"
server_key_iv: "56ddea69aae055be916bb59ac7d7ecb8"
server_key_version: "encryptionkey@0"
I20260619 18:43:01.300024 29946 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:43:01.302366 29961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:01.303748 29946 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:43:01.303886 29946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-2/wal
uuid: "5ee365d535f64fda9b04ac44dd08738c"
format_stamp: "Formatted at 2026-06-19 18:43:01 on dist-test-slave-mld5"
server_key: "eae4db4b7dbbd3f6ad3bef46ae4feb24"
server_key_iv: "56ddea69aae055be916bb59ac7d7ecb8"
server_key_version: "encryptionkey@0"
I20260619 18:43:01.304000 29946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:43:01.346730 29946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:43:01.350185 29946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:43:01.350390 29946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:43:01.352002 29946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:43:01.352072 29946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:01.352135 29946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:43:01.352171 29946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:43:01 dist-test-slave-mld5 krb5kdc[29549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894581, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM
I20260619 18:43:01.362574 29946 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.67:38273
I20260619 18:43:01.362684 30076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.67:38273 every 8 connection(s)
I20260619 18:43:01.363689 29946 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
I20260619 18:43:01.364804 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 29946
I20260619 18:43:01.364951 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance
I20260619 18:43:01.365262 22077 external_mini_cluster.cc:1428] Setting key c0cef1615791f9dc8711c56c8465c10e
I20260619 18:43:01.366959 29643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.352662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:46789 (local address 127.21.143.126:33147)
0619 18:43:01.352823 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:01.352828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:01.354142 (+ 1314us) server_negotiation.cc:407] Connection header received
0619 18:43:01.355697 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:01.355703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:01.355768 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:01.355869 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:01.357562 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.358303 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.359643 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.359884 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.359995 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.362869 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:01.362873 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:01.362878 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:01.362908 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:01.364613 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:01.365227 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:01.365231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:01.365234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:01.365329 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:01.365720 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:01.365727 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:01.365729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:01.365979 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:01.366105 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:01.366693 (+ 588us) server_negotiation.cc:299] Negotiation successful
0619 18:43:01.366810 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 18:43:01.367779 29970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.353493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33147 (local address 127.21.143.67:46789)
0619 18:43:01.353980 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:01.354040 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:01.355450 (+ 1410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:01.356104 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:01.356114 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:01.356637 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:01.357389 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:01.357410 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.358427 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.358431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:01.359523 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:01.359532 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.360032 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.360036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:01.360207 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.360811 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:01.360847 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:01.362662 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:43:01.364729 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:01.364735 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:01.364749 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:01.365126 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:01.365417 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:01.365420 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:01.365422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:01.365592 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:01.366114 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:43:01.366121 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:43:01.366421 (+ 300us) client_negotiation.cc:769] Sending connection context
0619 18:43:01.366621 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 18:43:01.366863 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":103,"threads_started":1}
I20260619 18:43:01.368834 30077 heartbeater.cc:347] Connected to a master server at 127.21.143.126:33147
I20260619 18:43:01.368960 30077 heartbeater.cc:464] Registering TS with master...
I20260619 18:43:01.369490 30077 heartbeater.cc:511] Master 127.21.143.126:33147 requested a full tablet report, sending...
I20260619 18:43:01.370579 29593 ts_manager.cc:194] Registered new tserver with Master: 5ee365d535f64fda9b04ac44dd08738c (127.21.143.67:38273)
I20260619 18:43:01.371367 29593 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.143.67@KRBTEST.COM'} at 127.21.143.67:46789
I20260619 18:43:01.377975 29643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.371246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:41799 (local address 127.21.143.126:33147)
0619 18:43:01.371384 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:01.371402 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:01.371526 (+ 124us) server_negotiation.cc:407] Connection header received
0619 18:43:01.371682 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:01.371686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:01.371728 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:01.371813 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:01.372311 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.372780 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.373690 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.373895 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.373975 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.374977 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:01.374979 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:01.374981 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:01.374997 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:01.376387 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:01.376785 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:01.376788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:01.376789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:01.376884 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:01.377186 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:01.377189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:01.377190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:01.377353 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:01.377427 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:01.377717 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 18:43:01.377824 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 18:43:01.378000 30083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.371169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33147 (local address 127.21.143.67:41799)
0619 18:43:01.371460 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:01.371474 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:01.371566 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:01.371845 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:01.371847 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:01.372081 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:01.372202 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:01.372209 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.372909 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.372912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:01.373593 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:01.373601 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.373991 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.373993 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:01.374059 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.374559 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:01.374572 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:01.374868 (+ 296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:43:01.376476 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:01.376479 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:01.376481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:01.376657 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:01.377009 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:01.377012 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:01.377013 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:01.377097 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:01.377441 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:43:01.377443 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:43:01.377518 (+ 75us) client_negotiation.cc:769] Sending connection context
0619 18:43:01.377656 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 18:43:01.377813 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":5,"thread_start_us":94,"threads_started":1}
I20260619 18:43:01.379256 22077 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 18:43:01 dist-test-slave-mld5 krb5kdc[29549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894581, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 18:43:01 dist-test-slave-mld5 krb5kdc[29549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:01 dist-test-slave-mld5 krb5kdc[29549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 18:43:01.402220 29643 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:01.396429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46496 (local address 127.21.143.126:33147)
0619 18:43:01.396563 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:01.396567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:01.396579 (+ 12us) server_negotiation.cc:407] Connection header received
0619 18:43:01.396624 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:01.396627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:01.396667 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:01.396742 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:01.397398 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.397907 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.398715 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.398921 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.398993 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.402099 (+ 3106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46496: BlockingRecv error: recv got EOF from 127.0.0.1:46496 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 18:43:01 dist-test-slave-mld5 krb5kdc[29549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894581, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM
I20260619 18:43:01.412076 29643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.403892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46512 (local address 127.21.143.126:33147)
0619 18:43:01.404053 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:01.404057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:01.404187 (+ 130us) server_negotiation.cc:407] Connection header received
0619 18:43:01.404341 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:01.404344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:01.404388 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:01.404481 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:01.405070 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.405590 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.406405 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.406620 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.406704 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.409312 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:01.409316 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:01.409318 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:01.409347 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:01.410569 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:01.410991 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:01.410994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:01.410995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:01.411065 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:01.411339 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:01.411343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:01.411344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:01.411532 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:01.411635 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:01.411853 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 18:43:01.411955 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 18:43:01.415012 29593 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:43:01.418578 29593 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:43:01.432994 30100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.427194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:36541 (local address 127.0.0.1:45454)
0619 18:43:01.427901 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:01.427917 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:01.428006 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:01.428805 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:01.428811 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:01.428835 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:43:01.429069 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:01.429079 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.430071 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.430077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:01.431517 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:01.431527 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.432443 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.432446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:01.432550 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.432584 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 18:43:01.432696 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 18:43:01.432817 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":111,"thread_start_us":288,"threads_started":1}
I20260619 18:43:01.433164 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.427106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:38273 (local address 127.0.0.1:43112)
0619 18:43:01.427444 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:01.427491 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:01.427612 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:01.428932 (+ 1320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:01.428935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:01.428967 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:43:01.429165 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:01.429173 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.430618 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.430622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:01.431760 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:01.431770 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.432739 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.432742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:01.432842 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.432866 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 18:43:01.432922 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 18:43:01.432991 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":44,"spinlock_wait_cycles":5632,"thread_start_us":88,"threads_started":1}
I20260619 18:43:01.433007 30102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.427836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:39881 (local address 127.0.0.1:55776)
0619 18:43:01.428210 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:01.428224 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:01.428318 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:01.428932 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:01.428935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:01.428968 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:43:01.429137 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:01.429147 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.430486 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.430489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:01.431657 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:01.431663 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.432705 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.432708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:01.432785 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.432805 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 18:43:01.432861 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 18:43:01.432914 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":49,"spinlock_wait_cycles":5120,"thread_start_us":97,"threads_started":1}
I20260619 18:43:01.433550 30103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.427824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43112 (local address 127.21.143.67:38273)
0619 18:43:01.428564 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:01.428574 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:01.428592 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:43:01.428653 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:01.428657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:01.428795 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:01.429294 (+ 499us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:43:01.429434 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.430505 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.431880 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.432602 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.433072 (+ 470us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.433235 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:01.433281 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 18:43:01.433395 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":100,"threads_started":1}
I20260619 18:43:01.433676 30104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.428020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55776 (local address 127.21.143.66:39881)
0619 18:43:01.428445 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:01.428456 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:01.428478 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:43:01.428552 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:01.428558 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:01.428754 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:01.429047 (+ 293us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:43:01.429229 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.430366 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.431763 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.432599 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.432717 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.432887 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:01.432939 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 18:43:01.433029 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":110,"threads_started":1}
I20260619 18:43:01.433707 30101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.427543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45454 (local address 127.21.143.65:36541)
0619 18:43:01.428321 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:01.428329 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:01.428352 (+ 23us) server_negotiation.cc:407] Connection header received
0619 18:43:01.428432 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:01.428438 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:01.428661 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:01.428831 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:43:01.429216 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.429958 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.431644 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.432305 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.432439 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.432601 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:01.433325 (+ 724us) server_negotiation.cc:299] Negotiation successful
0619 18:43:01.433551 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":86,"threads_started":1}
I20260619 18:43:01.436350 29731 tablet_service.cc:1467] Processing CreateTablet for tablet 9f431a8109b1450bb9c73461c91234d8 (DEFAULT_TABLE table=test-table [id=34c423f8104745098bedd1fd44617d46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:43:01.436357 29871 tablet_service.cc:1467] Processing CreateTablet for tablet 9f431a8109b1450bb9c73461c91234d8 (DEFAULT_TABLE table=test-table [id=34c423f8104745098bedd1fd44617d46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:43:01.436360 30011 tablet_service.cc:1467] Processing CreateTablet for tablet 9f431a8109b1450bb9c73461c91234d8 (DEFAULT_TABLE table=test-table [id=34c423f8104745098bedd1fd44617d46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:43:01.437353 29731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f431a8109b1450bb9c73461c91234d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:43:01.437367 29871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f431a8109b1450bb9c73461c91234d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:43:01.437377 30011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f431a8109b1450bb9c73461c91234d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:43:01.443676 30105 tablet_bootstrap.cc:492] T 9f431a8109b1450bb9c73461c91234d8 P 045f04c1759a4d9bbf38cc06b397ba76: Bootstrap starting.
I20260619 18:43:01.445318 30107 tablet_bootstrap.cc:492] T 9f431a8109b1450bb9c73461c91234d8 P b9d3e9ca17bd4d9babad18338ae019ee: Bootstrap starting.
I20260619 18:43:01.445318 30106 tablet_bootstrap.cc:492] T 9f431a8109b1450bb9c73461c91234d8 P 5ee365d535f64fda9b04ac44dd08738c: Bootstrap starting.
I20260619 18:43:01.446035 30105 tablet_bootstrap.cc:654] T 9f431a8109b1450bb9c73461c91234d8 P 045f04c1759a4d9bbf38cc06b397ba76: Neither blocks nor log segments found. Creating new log.
I20260619 18:43:01.446758 30105 log.cc:824] T 9f431a8109b1450bb9c73461c91234d8 P 045f04c1759a4d9bbf38cc06b397ba76: Log is configured to *not* fsync() on all Append() calls
I20260619 18:43:01.447438 30106 tablet_bootstrap.cc:654] T 9f431a8109b1450bb9c73461c91234d8 P 5ee365d535f64fda9b04ac44dd08738c: Neither blocks nor log segments found. Creating new log.
I20260619 18:43:01.447438 30107 tablet_bootstrap.cc:654] T 9f431a8109b1450bb9c73461c91234d8 P b9d3e9ca17bd4d9babad18338ae019ee: Neither blocks nor log segments found. Creating new log.
I20260619 18:43:01.448275 30107 log.cc:824] T 9f431a8109b1450bb9c73461c91234d8 P b9d3e9ca17bd4d9babad18338ae019ee: Log is configured to *not* fsync() on all Append() calls
I20260619 18:43:01.448274 30106 log.cc:824] T 9f431a8109b1450bb9c73461c91234d8 P 5ee365d535f64fda9b04ac44dd08738c: Log is configured to *not* fsync() on all Append() calls
I20260619 18:43:01.448742 30105 tablet_bootstrap.cc:492] T 9f431a8109b1450bb9c73461c91234d8 P 045f04c1759a4d9bbf38cc06b397ba76: No bootstrap required, opened a new log
I20260619 18:43:01.448956 30105 ts_tablet_manager.cc:1397] T 9f431a8109b1450bb9c73461c91234d8 P 045f04c1759a4d9bbf38cc06b397ba76: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 18:43:01.450125 30107 tablet_bootstrap.cc:492] T 9f431a8109b1450bb9c73461c91234d8 P b9d3e9ca17bd4d9babad18338ae019ee: No bootstrap required, opened a new log
I20260619 18:43:01.450125 30106 tablet_bootstrap.cc:492] T 9f431a8109b1450bb9c73461c91234d8 P 5ee365d535f64fda9b04ac44dd08738c: No bootstrap required, opened a new log
I20260619 18:43:01.450304 30107 ts_tablet_manager.cc:1397] T 9f431a8109b1450bb9c73461c91234d8 P b9d3e9ca17bd4d9babad18338ae019ee: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260619 18:43:01.450392 30106 ts_tablet_manager.cc:1397] T 9f431a8109b1450bb9c73461c91234d8 P 5ee365d535f64fda9b04ac44dd08738c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20260619 18:43:01.453569 30105 raft_consensus.cc:348] T 9f431a8109b1450bb9c73461c91234d8 P 045f04c1759a4d9bbf38cc06b397ba76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } } peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } }
I20260619 18:43:01.453821 30105 raft_consensus.cc:374] T 9f431a8109b1450bb9c73461c91234d8 P 045f04c1759a4d9bbf38cc06b397ba76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:43:01.453876 30105 raft_consensus.cc:729] T 9f431a8109b1450bb9c73461c91234d8 P 045f04c1759a4d9bbf38cc06b397ba76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 045f04c1759a4d9bbf38cc06b397ba76, State: Initialized, Role: FOLLOWER
I20260619 18:43:01.454319 30105 consensus_queue.cc:260] T 9f431a8109b1450bb9c73461c91234d8 P 045f04c1759a4d9bbf38cc06b397ba76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } } peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } }
I20260619 18:43:01.454296 30107 raft_consensus.cc:348] T 9f431a8109b1450bb9c73461c91234d8 P b9d3e9ca17bd4d9babad18338ae019ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } } peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } }
I20260619 18:43:01.454296 30106 raft_consensus.cc:348] T 9f431a8109b1450bb9c73461c91234d8 P 5ee365d535f64fda9b04ac44dd08738c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } } peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } }
I20260619 18:43:01.454542 30107 raft_consensus.cc:374] T 9f431a8109b1450bb9c73461c91234d8 P b9d3e9ca17bd4d9babad18338ae019ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:43:01.454542 30106 raft_consensus.cc:374] T 9f431a8109b1450bb9c73461c91234d8 P 5ee365d535f64fda9b04ac44dd08738c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:43:01.454603 30107 raft_consensus.cc:729] T 9f431a8109b1450bb9c73461c91234d8 P b9d3e9ca17bd4d9babad18338ae019ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d3e9ca17bd4d9babad18338ae019ee, State: Initialized, Role: FOLLOWER
I20260619 18:43:01.454603 30106 raft_consensus.cc:729] T 9f431a8109b1450bb9c73461c91234d8 P 5ee365d535f64fda9b04ac44dd08738c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee365d535f64fda9b04ac44dd08738c, State: Initialized, Role: FOLLOWER
I20260619 18:43:01.455018 30107 consensus_queue.cc:260] T 9f431a8109b1450bb9c73461c91234d8 P b9d3e9ca17bd4d9babad18338ae019ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } } peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } }
I20260619 18:43:01.455017 30106 consensus_queue.cc:260] T 9f431a8109b1450bb9c73461c91234d8 P 5ee365d535f64fda9b04ac44dd08738c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } } peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } }
I20260619 18:43:01.455382 30105 ts_tablet_manager.cc:1428] T 9f431a8109b1450bb9c73461c91234d8 P 045f04c1759a4d9bbf38cc06b397ba76: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260619 18:43:01.455724 29797 heartbeater.cc:503] Master 127.21.143.126:33147 was elected leader, sending a full tablet report...
I20260619 18:43:01.455961 30107 ts_tablet_manager.cc:1428] T 9f431a8109b1450bb9c73461c91234d8 P b9d3e9ca17bd4d9babad18338ae019ee: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260619 18:43:01.455961 30106 ts_tablet_manager.cc:1428] T 9f431a8109b1450bb9c73461c91234d8 P 5ee365d535f64fda9b04ac44dd08738c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 18:43:01.456281 30077 heartbeater.cc:503] Master 127.21.143.126:33147 was elected leader, sending a full tablet report...
I20260619 18:43:01.456300 29937 heartbeater.cc:503] Master 127.21.143.126:33147 was elected leader, sending a full tablet report...
W20260619 18:43:01.613926 29938 tablet.cc:2369] T 9f431a8109b1450bb9c73461c91234d8 P b9d3e9ca17bd4d9babad18338ae019ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:43:01.614620 30078 tablet.cc:2369] T 9f431a8109b1450bb9c73461c91234d8 P 5ee365d535f64fda9b04ac44dd08738c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:43:01.653980 29798 tablet.cc:2369] T 9f431a8109b1450bb9c73461c91234d8 P 045f04c1759a4d9bbf38cc06b397ba76: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:43:01.703591 30111 raft_consensus.cc:482] T 9f431a8109b1450bb9c73461c91234d8 P 045f04c1759a4d9bbf38cc06b397ba76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:43:01.703841 30111 raft_consensus.cc:504] T 9f431a8109b1450bb9c73461c91234d8 P 045f04c1759a4d9bbf38cc06b397ba76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } } peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } }
I20260619 18:43:01.705015 30111 leader_election.cc:288] T 9f431a8109b1450bb9c73461c91234d8 P 045f04c1759a4d9bbf38cc06b397ba76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ee365d535f64fda9b04ac44dd08738c (127.21.143.67:38273), b9d3e9ca17bd4d9babad18338ae019ee (127.21.143.66:39881)
I20260619 18:43:01.710333 30103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.705506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:34371 (local address 127.21.143.67:38273)
0619 18:43:01.705656 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:01.705659 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:01.705895 (+ 236us) server_negotiation.cc:407] Connection header received
0619 18:43:01.705997 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:01.706000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:01.706053 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:01.706139 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:43:01.706562 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.707405 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.709309 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.709703 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.709840 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.709898 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:01.710120 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 18:43:01.710201 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 18:43:01.710341 30104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.705376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:45799 (local address 127.21.143.66:39881)
0619 18:43:01.705527 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:01.705533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:01.705837 (+ 304us) server_negotiation.cc:407] Connection header received
0619 18:43:01.706001 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:01.706003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:01.706053 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:01.706139 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:43:01.706562 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.707413 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.709316 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.709727 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.709867 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.709924 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:01.710135 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 18:43:01.710213 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 18:43:01.710364 30114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.705278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:38273 (local address 127.21.143.65:34371)
0619 18:43:01.705809 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:01.705821 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:01.705897 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:01.706164 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:01.706167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:01.706208 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:43:01.706443 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:01.706451 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.707526 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.707530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:01.709148 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:01.709159 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.709837 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.709841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:01.709953 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.710000 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 18:43:01.710088 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 18:43:01.710182 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":327,"spinlock_wait_cycles":7424,"thread_start_us":87,"threads_started":1}
I20260619 18:43:01.710371 30115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.705278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:39881 (local address 127.21.143.65:45799)
0619 18:43:01.705721 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:01.705737 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:01.705883 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:01.706165 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:01.706167 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:01.706207 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:43:01.706443 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:01.706451 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.707526 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.707530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:01.709148 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:01.709159 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.709867 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.709871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:01.710002 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.710028 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 18:43:01.710116 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 18:43:01.710193 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":196,"spinlock_wait_cycles":29696,"thread_start_us":107,"threads_started":1}
I20260619 18:43:01.711159 30031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f431a8109b1450bb9c73461c91234d8" candidate_uuid: "045f04c1759a4d9bbf38cc06b397ba76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ee365d535f64fda9b04ac44dd08738c" is_pre_election: true
I20260619 18:43:01.711422 29891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f431a8109b1450bb9c73461c91234d8" candidate_uuid: "045f04c1759a4d9bbf38cc06b397ba76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" is_pre_election: true
I20260619 18:43:01.711511 30031 raft_consensus.cc:2454] T 9f431a8109b1450bb9c73461c91234d8 P 5ee365d535f64fda9b04ac44dd08738c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 045f04c1759a4d9bbf38cc06b397ba76 in term 0.
I20260619 18:43:01.711719 29891 raft_consensus.cc:2454] T 9f431a8109b1450bb9c73461c91234d8 P b9d3e9ca17bd4d9babad18338ae019ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 045f04c1759a4d9bbf38cc06b397ba76 in term 0.
I20260619 18:43:01.712086 29682 leader_election.cc:302] T 9f431a8109b1450bb9c73461c91234d8 P 045f04c1759a4d9bbf38cc06b397ba76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 045f04c1759a4d9bbf38cc06b397ba76, 5ee365d535f64fda9b04ac44dd08738c; no voters:
I20260619 18:43:01.712363 30111 raft_consensus.cc:2781] T 9f431a8109b1450bb9c73461c91234d8 P 045f04c1759a4d9bbf38cc06b397ba76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:43:01.712445 30111 raft_consensus.cc:482] T 9f431a8109b1450bb9c73461c91234d8 P 045f04c1759a4d9bbf38cc06b397ba76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:43:01.712486 30111 raft_consensus.cc:3037] T 9f431a8109b1450bb9c73461c91234d8 P 045f04c1759a4d9bbf38cc06b397ba76 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:43:01.713486 30111 raft_consensus.cc:504] T 9f431a8109b1450bb9c73461c91234d8 P 045f04c1759a4d9bbf38cc06b397ba76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } } peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } }
I20260619 18:43:01.713879 30111 leader_election.cc:288] T 9f431a8109b1450bb9c73461c91234d8 P 045f04c1759a4d9bbf38cc06b397ba76 [CANDIDATE]: Term 1 election: Requested vote from peers 5ee365d535f64fda9b04ac44dd08738c (127.21.143.67:38273), b9d3e9ca17bd4d9babad18338ae019ee (127.21.143.66:39881)
I20260619 18:43:01.714207 30031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f431a8109b1450bb9c73461c91234d8" candidate_uuid: "045f04c1759a4d9bbf38cc06b397ba76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ee365d535f64fda9b04ac44dd08738c"
I20260619 18:43:01.714287 29891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f431a8109b1450bb9c73461c91234d8" candidate_uuid: "045f04c1759a4d9bbf38cc06b397ba76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d3e9ca17bd4d9babad18338ae019ee"
I20260619 18:43:01.714342 30031 raft_consensus.cc:3037] T 9f431a8109b1450bb9c73461c91234d8 P 5ee365d535f64fda9b04ac44dd08738c [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:43:01.714424 29891 raft_consensus.cc:3037] T 9f431a8109b1450bb9c73461c91234d8 P b9d3e9ca17bd4d9babad18338ae019ee [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:43:01.715459 29891 raft_consensus.cc:2454] T 9f431a8109b1450bb9c73461c91234d8 P b9d3e9ca17bd4d9babad18338ae019ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 045f04c1759a4d9bbf38cc06b397ba76 in term 1.
I20260619 18:43:01.715749 29684 leader_election.cc:302] T 9f431a8109b1450bb9c73461c91234d8 P 045f04c1759a4d9bbf38cc06b397ba76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 045f04c1759a4d9bbf38cc06b397ba76, b9d3e9ca17bd4d9babad18338ae019ee; no voters:
I20260619 18:43:01.715914 30031 raft_consensus.cc:2454] T 9f431a8109b1450bb9c73461c91234d8 P 5ee365d535f64fda9b04ac44dd08738c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 045f04c1759a4d9bbf38cc06b397ba76 in term 1.
I20260619 18:43:01.715921 30111 raft_consensus.cc:2781] T 9f431a8109b1450bb9c73461c91234d8 P 045f04c1759a4d9bbf38cc06b397ba76 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:43:01.716202 30111 raft_consensus.cc:686] T 9f431a8109b1450bb9c73461c91234d8 P 045f04c1759a4d9bbf38cc06b397ba76 [term 1 LEADER]: Becoming Leader. State: Replica: 045f04c1759a4d9bbf38cc06b397ba76, State: Running, Role: LEADER
I20260619 18:43:01.716516 30111 consensus_queue.cc:237] T 9f431a8109b1450bb9c73461c91234d8 P 045f04c1759a4d9bbf38cc06b397ba76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } } peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } }
I20260619 18:43:01.719686 29593 catalog_manager.cc:5495] T 9f431a8109b1450bb9c73461c91234d8 P 045f04c1759a4d9bbf38cc06b397ba76 reported cstate change: term changed from 0 to 1, leader changed from <none> to 045f04c1759a4d9bbf38cc06b397ba76 (127.21.143.65). New cstate: current_term: 1 leader_uuid: "045f04c1759a4d9bbf38cc06b397ba76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } health_report { overall_health: UNKNOWN } } }
I20260619 18:43:01.793327 30101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.789523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45468 (local address 127.21.143.65:36541)
0619 18:43:01.789677 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:01.789681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:01.789696 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:43:01.789877 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:01.789880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:01.789944 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:01.790055 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:43:01.790362 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.790885 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.791809 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.792036 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.792161 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.792454 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:43:01.792918 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:43:01.793051 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:01.793113 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 18:43:01.793172 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 18:43:01.799384 29891 raft_consensus.cc:1261] T 9f431a8109b1450bb9c73461c91234d8 P b9d3e9ca17bd4d9babad18338ae019ee [term 1 FOLLOWER]: Refusing update from remote peer 045f04c1759a4d9bbf38cc06b397ba76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:43:01.799400 30031 raft_consensus.cc:1261] T 9f431a8109b1450bb9c73461c91234d8 P 5ee365d535f64fda9b04ac44dd08738c [term 1 FOLLOWER]: Refusing update from remote peer 045f04c1759a4d9bbf38cc06b397ba76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:43:01.800072 30111 consensus_queue.cc:1035] T 9f431a8109b1450bb9c73461c91234d8 P 045f04c1759a4d9bbf38cc06b397ba76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:43:01.800351 30116 consensus_queue.cc:1035] T 9f431a8109b1450bb9c73461c91234d8 P 045f04c1759a4d9bbf38cc06b397ba76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:43:01.809077 30120 mvcc.cc:205] Tried to move back new op lower bound from 7298640207042613248 to 7298640206713483264. Current Snapshot: MvccSnapshot[applied={T|T < 7298640207042613248}]
I20260619 18:43:01.809293 30119 mvcc.cc:205] Tried to move back new op lower bound from 7298640207042613248 to 7298640206713483264. Current Snapshot: MvccSnapshot[applied={T|T < 7298640207042613248}]
I20260619 18:43:01.818542 29593 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:43:01.818732 29593 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:43:01.821146 29593 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3f24326df2f48d7bff275e80e8cbcfe: Sending DeleteTablet for 3 replicas of tablet 9f431a8109b1450bb9c73461c91234d8
I20260619 18:43:01.821918 30011 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f431a8109b1450bb9c73461c91234d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:43:01 UTC) from {username='oryx', principal='oryx/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:43112
I20260619 18:43:01.822031 29871 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f431a8109b1450bb9c73461c91234d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:43:01 UTC) from {username='oryx', principal='oryx/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:55776
I20260619 18:43:01.822036 29731 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f431a8109b1450bb9c73461c91234d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:43:01 UTC) from {username='oryx', principal='oryx/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:45454
I20260619 18:43:01.822424 30131 tablet_replica.cc:330] T 9f431a8109b1450bb9c73461c91234d8 P 5ee365d535f64fda9b04ac44dd08738c: stopping tablet replica
I20260619 18:43:01.822737 30131 raft_consensus.cc:2229] T 9f431a8109b1450bb9c73461c91234d8 P 5ee365d535f64fda9b04ac44dd08738c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:43:01.823083 30131 raft_consensus.cc:2258] T 9f431a8109b1450bb9c73461c91234d8 P 5ee365d535f64fda9b04ac44dd08738c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:43:01.823119 30132 tablet_replica.cc:330] T 9f431a8109b1450bb9c73461c91234d8 P b9d3e9ca17bd4d9babad18338ae019ee: stopping tablet replica
I20260619 18:43:01.823562 30132 raft_consensus.cc:2229] T 9f431a8109b1450bb9c73461c91234d8 P b9d3e9ca17bd4d9babad18338ae019ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:43:01.823839 30132 raft_consensus.cc:2258] T 9f431a8109b1450bb9c73461c91234d8 P b9d3e9ca17bd4d9babad18338ae019ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:43:01.823238 29593 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:43:01.824232 29593 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:43:01.824632 30131 ts_tablet_manager.cc:1905] T 9f431a8109b1450bb9c73461c91234d8 P 5ee365d535f64fda9b04ac44dd08738c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:43:01.825273 30132 ts_tablet_manager.cc:1905] T 9f431a8109b1450bb9c73461c91234d8 P b9d3e9ca17bd4d9babad18338ae019ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:43:01.827633 30131 ts_tablet_manager.cc:1918] T 9f431a8109b1450bb9c73461c91234d8 P 5ee365d535f64fda9b04ac44dd08738c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:43:01.827731 30131 log.cc:1196] T 9f431a8109b1450bb9c73461c91234d8 P 5ee365d535f64fda9b04ac44dd08738c: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-2/wal/wals/9f431a8109b1450bb9c73461c91234d8
I20260619 18:43:01.828075 30131 ts_tablet_manager.cc:1939] T 9f431a8109b1450bb9c73461c91234d8 P 5ee365d535f64fda9b04ac44dd08738c: Deleting consensus metadata
I20260619 18:43:01.828052 30132 ts_tablet_manager.cc:1918] T 9f431a8109b1450bb9c73461c91234d8 P b9d3e9ca17bd4d9babad18338ae019ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:43:01.828142 30132 log.cc:1196] T 9f431a8109b1450bb9c73461c91234d8 P b9d3e9ca17bd4d9babad18338ae019ee: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-1/wal/wals/9f431a8109b1450bb9c73461c91234d8
I20260619 18:43:01.828441 30132 ts_tablet_manager.cc:1939] T 9f431a8109b1450bb9c73461c91234d8 P b9d3e9ca17bd4d9babad18338ae019ee: Deleting consensus metadata
I20260619 18:43:01.829464 29578 catalog_manager.cc:4841] TS 5ee365d535f64fda9b04ac44dd08738c (127.21.143.67:38273): tablet 9f431a8109b1450bb9c73461c91234d8 (table test-table [id=34c423f8104745098bedd1fd44617d46]) successfully deleted
I20260619 18:43:01.829502 30133 tablet_replica.cc:330] T 9f431a8109b1450bb9c73461c91234d8 P 045f04c1759a4d9bbf38cc06b397ba76: stopping tablet replica
I20260619 18:43:01.829787 30133 raft_consensus.cc:2229] T 9f431a8109b1450bb9c73461c91234d8 P 045f04c1759a4d9bbf38cc06b397ba76 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:43:01.830235 30133 raft_consensus.cc:2258] T 9f431a8109b1450bb9c73461c91234d8 P 045f04c1759a4d9bbf38cc06b397ba76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:43:01.830778 29871 tablet_service.cc:1467] Processing CreateTablet for tablet 2921c09567e64765ab5576f2f192216f (DEFAULT_TABLE table=test-table [id=5edb2059fe624ab396438019b4e4fe57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:43:01.830806 30011 tablet_service.cc:1467] Processing CreateTablet for tablet 2921c09567e64765ab5576f2f192216f (DEFAULT_TABLE table=test-table [id=5edb2059fe624ab396438019b4e4fe57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:43:01.831038 29871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2921c09567e64765ab5576f2f192216f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:43:01.831038 30011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2921c09567e64765ab5576f2f192216f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:43:01.831672 30133 ts_tablet_manager.cc:1905] T 9f431a8109b1450bb9c73461c91234d8 P 045f04c1759a4d9bbf38cc06b397ba76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:43:01.833222 30106 tablet_bootstrap.cc:492] T 2921c09567e64765ab5576f2f192216f P 5ee365d535f64fda9b04ac44dd08738c: Bootstrap starting.
I20260619 18:43:01.833525 29731 tablet_service.cc:1467] Processing CreateTablet for tablet 2921c09567e64765ab5576f2f192216f (DEFAULT_TABLE table=test-table [id=5edb2059fe624ab396438019b4e4fe57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:43:01.833762 29731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2921c09567e64765ab5576f2f192216f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:43:01.834031 30133 ts_tablet_manager.cc:1918] T 9f431a8109b1450bb9c73461c91234d8 P 045f04c1759a4d9bbf38cc06b397ba76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:43:01.834116 30133 log.cc:1196] T 9f431a8109b1450bb9c73461c91234d8 P 045f04c1759a4d9bbf38cc06b397ba76: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-0/wal/wals/9f431a8109b1450bb9c73461c91234d8
I20260619 18:43:01.834221 30106 tablet_bootstrap.cc:654] T 2921c09567e64765ab5576f2f192216f P 5ee365d535f64fda9b04ac44dd08738c: Neither blocks nor log segments found. Creating new log.
I20260619 18:43:01.834444 30133 ts_tablet_manager.cc:1939] T 9f431a8109b1450bb9c73461c91234d8 P 045f04c1759a4d9bbf38cc06b397ba76: Deleting consensus metadata
I20260619 18:43:01.834856 30107 tablet_bootstrap.cc:492] T 2921c09567e64765ab5576f2f192216f P b9d3e9ca17bd4d9babad18338ae019ee: Bootstrap starting.
I20260619 18:43:01.835696 29580 catalog_manager.cc:4841] TS 045f04c1759a4d9bbf38cc06b397ba76 (127.21.143.65:36541): tablet 9f431a8109b1450bb9c73461c91234d8 (table test-table [id=34c423f8104745098bedd1fd44617d46]) successfully deleted
I20260619 18:43:01.835807 30107 tablet_bootstrap.cc:654] T 2921c09567e64765ab5576f2f192216f P b9d3e9ca17bd4d9babad18338ae019ee: Neither blocks nor log segments found. Creating new log.
I20260619 18:43:01.836104 29580 catalog_manager.cc:4841] TS b9d3e9ca17bd4d9babad18338ae019ee (127.21.143.66:39881): tablet 9f431a8109b1450bb9c73461c91234d8 (table test-table [id=34c423f8104745098bedd1fd44617d46]) successfully deleted
I20260619 18:43:01.836858 30105 tablet_bootstrap.cc:492] T 2921c09567e64765ab5576f2f192216f P 045f04c1759a4d9bbf38cc06b397ba76: Bootstrap starting.
I20260619 18:43:01.837201 30106 tablet_bootstrap.cc:492] T 2921c09567e64765ab5576f2f192216f P 5ee365d535f64fda9b04ac44dd08738c: No bootstrap required, opened a new log
I20260619 18:43:01.837276 30106 ts_tablet_manager.cc:1397] T 2921c09567e64765ab5576f2f192216f P 5ee365d535f64fda9b04ac44dd08738c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 18:43:01.837508 30107 tablet_bootstrap.cc:492] T 2921c09567e64765ab5576f2f192216f P b9d3e9ca17bd4d9babad18338ae019ee: No bootstrap required, opened a new log
I20260619 18:43:01.837582 30107 ts_tablet_manager.cc:1397] T 2921c09567e64765ab5576f2f192216f P b9d3e9ca17bd4d9babad18338ae019ee: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 18:43:01.837719 30106 raft_consensus.cc:348] T 2921c09567e64765ab5576f2f192216f P 5ee365d535f64fda9b04ac44dd08738c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } }
I20260619 18:43:01.837833 30105 tablet_bootstrap.cc:654] T 2921c09567e64765ab5576f2f192216f P 045f04c1759a4d9bbf38cc06b397ba76: Neither blocks nor log segments found. Creating new log.
I20260619 18:43:01.837841 30106 raft_consensus.cc:374] T 2921c09567e64765ab5576f2f192216f P 5ee365d535f64fda9b04ac44dd08738c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:43:01.837877 30106 raft_consensus.cc:729] T 2921c09567e64765ab5576f2f192216f P 5ee365d535f64fda9b04ac44dd08738c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee365d535f64fda9b04ac44dd08738c, State: Initialized, Role: FOLLOWER
I20260619 18:43:01.837998 30106 consensus_queue.cc:260] T 2921c09567e64765ab5576f2f192216f P 5ee365d535f64fda9b04ac44dd08738c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } }
I20260619 18:43:01.838227 30106 ts_tablet_manager.cc:1428] T 2921c09567e64765ab5576f2f192216f P 5ee365d535f64fda9b04ac44dd08738c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:43:01.838064 30107 raft_consensus.cc:348] T 2921c09567e64765ab5576f2f192216f P b9d3e9ca17bd4d9babad18338ae019ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } }
I20260619 18:43:01.839131 30107 raft_consensus.cc:374] T 2921c09567e64765ab5576f2f192216f P b9d3e9ca17bd4d9babad18338ae019ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:43:01.839180 30107 raft_consensus.cc:729] T 2921c09567e64765ab5576f2f192216f P b9d3e9ca17bd4d9babad18338ae019ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d3e9ca17bd4d9babad18338ae019ee, State: Initialized, Role: FOLLOWER
I20260619 18:43:01.839370 30107 consensus_queue.cc:260] T 2921c09567e64765ab5576f2f192216f P b9d3e9ca17bd4d9babad18338ae019ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } }
I20260619 18:43:01.839432 30105 tablet_bootstrap.cc:492] T 2921c09567e64765ab5576f2f192216f P 045f04c1759a4d9bbf38cc06b397ba76: No bootstrap required, opened a new log
I20260619 18:43:01.839507 30105 ts_tablet_manager.cc:1397] T 2921c09567e64765ab5576f2f192216f P 045f04c1759a4d9bbf38cc06b397ba76: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 18:43:01.839725 30107 ts_tablet_manager.cc:1428] T 2921c09567e64765ab5576f2f192216f P b9d3e9ca17bd4d9babad18338ae019ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 18:43:01.839985 30105 raft_consensus.cc:348] T 2921c09567e64765ab5576f2f192216f P 045f04c1759a4d9bbf38cc06b397ba76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } }
I20260619 18:43:01.840088 30105 raft_consensus.cc:374] T 2921c09567e64765ab5576f2f192216f P 045f04c1759a4d9bbf38cc06b397ba76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:43:01.840119 30105 raft_consensus.cc:729] T 2921c09567e64765ab5576f2f192216f P 045f04c1759a4d9bbf38cc06b397ba76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 045f04c1759a4d9bbf38cc06b397ba76, State: Initialized, Role: FOLLOWER
I20260619 18:43:01.840245 30105 consensus_queue.cc:260] T 2921c09567e64765ab5576f2f192216f P 045f04c1759a4d9bbf38cc06b397ba76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } }
I20260619 18:43:01.840524 30105 ts_tablet_manager.cc:1428] T 2921c09567e64765ab5576f2f192216f P 045f04c1759a4d9bbf38cc06b397ba76: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 18:43:02.170265 30113 raft_consensus.cc:482] T 2921c09567e64765ab5576f2f192216f P 5ee365d535f64fda9b04ac44dd08738c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:43:02.170463 30113 raft_consensus.cc:504] T 2921c09567e64765ab5576f2f192216f P 5ee365d535f64fda9b04ac44dd08738c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } }
I20260619 18:43:02.171937 30113 leader_election.cc:288] T 2921c09567e64765ab5576f2f192216f P 5ee365d535f64fda9b04ac44dd08738c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 045f04c1759a4d9bbf38cc06b397ba76 (127.21.143.65:36541), b9d3e9ca17bd4d9babad18338ae019ee (127.21.143.66:39881)
I20260619 18:43:02.176919 30101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:02.172343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:44965 (local address 127.21.143.65:36541)
0619 18:43:02.172491 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:02.172496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:02.172634 (+ 138us) server_negotiation.cc:407] Connection header received
0619 18:43:02.172825 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:02.172828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:02.172881 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:02.172967 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:43:02.173286 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:02.174001 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:02.175528 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:02.176195 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:02.176312 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:02.176363 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:02.176722 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 18:43:02.176792 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 18:43:02.176919 30104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:02.173112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:42187 (local address 127.21.143.66:39881)
0619 18:43:02.173250 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:02.173257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:02.173276 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:43:02.173376 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:02.173380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:02.173443 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:02.173539 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:43:02.173811 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:02.174311 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:02.175657 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:02.176206 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:02.176311 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:02.176363 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:02.176723 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 18:43:02.176792 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:43:02.177071 30137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:02.172259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:36541 (local address 127.21.143.67:44965)
0619 18:43:02.172528 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:02.172543 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:02.172691 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:02.172980 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:02.172983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:02.172998 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:43:02.173146 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:02.173155 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:02.174109 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:02.174111 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:02.175389 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:02.175408 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:02.176318 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:02.176322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:02.176559 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:02.176619 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 18:43:02.176693 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 18:43:02.176836 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":115,"thread_start_us":97,"threads_started":1}
I20260619 18:43:02.177392 30138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:02.172701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:39881 (local address 127.21.143.67:42187)
0619 18:43:02.173196 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:02.173210 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:02.173281 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:02.173556 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:02.173559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:02.173576 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:43:02.173669 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:02.173675 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:02.174450 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:02.174453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:02.175554 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:02.175561 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:02.176318 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:02.176322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:02.176507 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:02.176617 (+ 110us) client_negotiation.cc:769] Sending connection context
0619 18:43:02.176693 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 18:43:02.176836 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":120,"thread_start_us":78,"threads_started":1}
I20260619 18:43:02.177623 29751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2921c09567e64765ab5576f2f192216f" candidate_uuid: "5ee365d535f64fda9b04ac44dd08738c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "045f04c1759a4d9bbf38cc06b397ba76" is_pre_election: true
I20260619 18:43:02.177824 29751 raft_consensus.cc:2454] T 2921c09567e64765ab5576f2f192216f P 045f04c1759a4d9bbf38cc06b397ba76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ee365d535f64fda9b04ac44dd08738c in term 0.
I20260619 18:43:02.178242 29964 leader_election.cc:302] T 2921c09567e64765ab5576f2f192216f P 5ee365d535f64fda9b04ac44dd08738c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 045f04c1759a4d9bbf38cc06b397ba76, 5ee365d535f64fda9b04ac44dd08738c; no voters:
I20260619 18:43:02.178349 29891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2921c09567e64765ab5576f2f192216f" candidate_uuid: "5ee365d535f64fda9b04ac44dd08738c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" is_pre_election: true
I20260619 18:43:02.178480 29891 raft_consensus.cc:2454] T 2921c09567e64765ab5576f2f192216f P b9d3e9ca17bd4d9babad18338ae019ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ee365d535f64fda9b04ac44dd08738c in term 0.
I20260619 18:43:02.178521 30113 raft_consensus.cc:2781] T 2921c09567e64765ab5576f2f192216f P 5ee365d535f64fda9b04ac44dd08738c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:43:02.178567 30113 raft_consensus.cc:482] T 2921c09567e64765ab5576f2f192216f P 5ee365d535f64fda9b04ac44dd08738c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:43:02.178601 30113 raft_consensus.cc:3037] T 2921c09567e64765ab5576f2f192216f P 5ee365d535f64fda9b04ac44dd08738c [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:43:02.179752 30113 raft_consensus.cc:504] T 2921c09567e64765ab5576f2f192216f P 5ee365d535f64fda9b04ac44dd08738c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } }
I20260619 18:43:02.180080 30113 leader_election.cc:288] T 2921c09567e64765ab5576f2f192216f P 5ee365d535f64fda9b04ac44dd08738c [CANDIDATE]: Term 1 election: Requested vote from peers 045f04c1759a4d9bbf38cc06b397ba76 (127.21.143.65:36541), b9d3e9ca17bd4d9babad18338ae019ee (127.21.143.66:39881)
I20260619 18:43:02.180519 29751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2921c09567e64765ab5576f2f192216f" candidate_uuid: "5ee365d535f64fda9b04ac44dd08738c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "045f04c1759a4d9bbf38cc06b397ba76"
I20260619 18:43:02.180516 29891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2921c09567e64765ab5576f2f192216f" candidate_uuid: "5ee365d535f64fda9b04ac44dd08738c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d3e9ca17bd4d9babad18338ae019ee"
I20260619 18:43:02.180644 29751 raft_consensus.cc:3037] T 2921c09567e64765ab5576f2f192216f P 045f04c1759a4d9bbf38cc06b397ba76 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:43:02.180644 29891 raft_consensus.cc:3037] T 2921c09567e64765ab5576f2f192216f P b9d3e9ca17bd4d9babad18338ae019ee [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:43:02.181713 29891 raft_consensus.cc:2454] T 2921c09567e64765ab5576f2f192216f P b9d3e9ca17bd4d9babad18338ae019ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ee365d535f64fda9b04ac44dd08738c in term 1.
I20260619 18:43:02.181774 29751 raft_consensus.cc:2454] T 2921c09567e64765ab5576f2f192216f P 045f04c1759a4d9bbf38cc06b397ba76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ee365d535f64fda9b04ac44dd08738c in term 1.
I20260619 18:43:02.182011 29964 leader_election.cc:302] T 2921c09567e64765ab5576f2f192216f P 5ee365d535f64fda9b04ac44dd08738c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ee365d535f64fda9b04ac44dd08738c, b9d3e9ca17bd4d9babad18338ae019ee; no voters:
I20260619 18:43:02.182170 30113 raft_consensus.cc:2781] T 2921c09567e64765ab5576f2f192216f P 5ee365d535f64fda9b04ac44dd08738c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:43:02.182348 30113 raft_consensus.cc:686] T 2921c09567e64765ab5576f2f192216f P 5ee365d535f64fda9b04ac44dd08738c [term 1 LEADER]: Becoming Leader. State: Replica: 5ee365d535f64fda9b04ac44dd08738c, State: Running, Role: LEADER
I20260619 18:43:02.182600 30113 consensus_queue.cc:237] T 2921c09567e64765ab5576f2f192216f P 5ee365d535f64fda9b04ac44dd08738c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } }
I20260619 18:43:02.184825 29594 catalog_manager.cc:5495] T 2921c09567e64765ab5576f2f192216f P 5ee365d535f64fda9b04ac44dd08738c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ee365d535f64fda9b04ac44dd08738c (127.21.143.67). New cstate: current_term: 1 leader_uuid: "5ee365d535f64fda9b04ac44dd08738c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } health_report { overall_health: HEALTHY } } }
Jun 19 18:43:02 dist-test-slave-mld5 krb5kdc[29549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894580, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.126@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM
I20260619 18:43:02.236965 30146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:02.229061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33147 (local address 127.0.0.1:46514)
0619 18:43:02.229375 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:02.229388 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:02.229494 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:02.229760 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:02.229762 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:02.230158 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:02.230251 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:02.230257 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:02.230978 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:02.230980 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:02.231602 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:02.231611 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:02.232032 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:02.232034 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:02.232091 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:02.232605 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:02.232616 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:02.233982 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:43:02.235526 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:02.235533 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:02.235538 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:02.235763 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:02.236056 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:02.236058 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:02.236059 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:02.236136 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:02.236496 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:43:02.236502 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:43:02.236616 (+ 114us) client_negotiation.cc:769] Sending connection context
0619 18:43:02.236715 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 18:43:02.236824 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":91,"threads_started":1}
I20260619 18:43:02.236999 30147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:02.229146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46514 (local address 127.21.143.126:33147)
0619 18:43:02.229411 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:02.229427 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:02.229445 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:43:02.229597 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:02.229599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:02.229641 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:02.229718 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:02.230354 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:02.230880 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:02.231708 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:02.231939 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:02.232015 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:02.234102 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:02.234105 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:02.234107 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:02.234127 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:02.235425 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:02.235880 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:02.235883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:02.235885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:02.235966 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:02.236215 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:02.236218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:02.236219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:02.236398 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:02.236496 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:02.236778 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 18:43:02.236882 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":122,"threads_started":1}
I20260619 18:43:02.240832 29594 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:46514:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 18:43:02.241619 29594 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:43:02.246191 29731 tablet_service.cc:1467] Processing CreateTablet for tablet 0caa10cb2bee455eb70a3dae38460c4d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=33558411789a4688a0feec7e88cc4e9b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:43:02.246225 29871 tablet_service.cc:1467] Processing CreateTablet for tablet 0caa10cb2bee455eb70a3dae38460c4d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=33558411789a4688a0feec7e88cc4e9b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:43:02.246584 29871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0caa10cb2bee455eb70a3dae38460c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:43:02.246583 29731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0caa10cb2bee455eb70a3dae38460c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:43:02.246774 30011 tablet_service.cc:1467] Processing CreateTablet for tablet 0caa10cb2bee455eb70a3dae38460c4d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=33558411789a4688a0feec7e88cc4e9b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:43:02.247148 30011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0caa10cb2bee455eb70a3dae38460c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:43:02.249132 30107 tablet_bootstrap.cc:492] T 0caa10cb2bee455eb70a3dae38460c4d P b9d3e9ca17bd4d9babad18338ae019ee: Bootstrap starting.
I20260619 18:43:02.249508 30105 tablet_bootstrap.cc:492] T 0caa10cb2bee455eb70a3dae38460c4d P 045f04c1759a4d9bbf38cc06b397ba76: Bootstrap starting.
I20260619 18:43:02.250052 30106 tablet_bootstrap.cc:492] T 0caa10cb2bee455eb70a3dae38460c4d P 5ee365d535f64fda9b04ac44dd08738c: Bootstrap starting.
I20260619 18:43:02.250093 30107 tablet_bootstrap.cc:654] T 0caa10cb2bee455eb70a3dae38460c4d P b9d3e9ca17bd4d9babad18338ae019ee: Neither blocks nor log segments found. Creating new log.
I20260619 18:43:02.250362 30105 tablet_bootstrap.cc:654] T 0caa10cb2bee455eb70a3dae38460c4d P 045f04c1759a4d9bbf38cc06b397ba76: Neither blocks nor log segments found. Creating new log.
I20260619 18:43:02.251009 30106 tablet_bootstrap.cc:654] T 0caa10cb2bee455eb70a3dae38460c4d P 5ee365d535f64fda9b04ac44dd08738c: Neither blocks nor log segments found. Creating new log.
I20260619 18:43:02.251662 30105 tablet_bootstrap.cc:492] T 0caa10cb2bee455eb70a3dae38460c4d P 045f04c1759a4d9bbf38cc06b397ba76: No bootstrap required, opened a new log
I20260619 18:43:02.251739 30107 tablet_bootstrap.cc:492] T 0caa10cb2bee455eb70a3dae38460c4d P b9d3e9ca17bd4d9babad18338ae019ee: No bootstrap required, opened a new log
I20260619 18:43:02.251752 30105 ts_tablet_manager.cc:1397] T 0caa10cb2bee455eb70a3dae38460c4d P 045f04c1759a4d9bbf38cc06b397ba76: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 18:43:02.251820 30107 ts_tablet_manager.cc:1397] T 0caa10cb2bee455eb70a3dae38460c4d P b9d3e9ca17bd4d9babad18338ae019ee: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 18:43:02.252197 30106 tablet_bootstrap.cc:492] T 0caa10cb2bee455eb70a3dae38460c4d P 5ee365d535f64fda9b04ac44dd08738c: No bootstrap required, opened a new log
I20260619 18:43:02.252153 30105 raft_consensus.cc:348] T 0caa10cb2bee455eb70a3dae38460c4d P 045f04c1759a4d9bbf38cc06b397ba76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } }
I20260619 18:43:02.252269 30105 raft_consensus.cc:374] T 0caa10cb2bee455eb70a3dae38460c4d P 045f04c1759a4d9bbf38cc06b397ba76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:43:02.252221 30107 raft_consensus.cc:348] T 0caa10cb2bee455eb70a3dae38460c4d P b9d3e9ca17bd4d9babad18338ae019ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } }
I20260619 18:43:02.252310 30106 ts_tablet_manager.cc:1397] T 0caa10cb2bee455eb70a3dae38460c4d P 5ee365d535f64fda9b04ac44dd08738c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 18:43:02.252308 30105 raft_consensus.cc:729] T 0caa10cb2bee455eb70a3dae38460c4d P 045f04c1759a4d9bbf38cc06b397ba76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 045f04c1759a4d9bbf38cc06b397ba76, State: Initialized, Role: FOLLOWER
I20260619 18:43:02.252321 30107 raft_consensus.cc:374] T 0caa10cb2bee455eb70a3dae38460c4d P b9d3e9ca17bd4d9babad18338ae019ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:43:02.252357 30107 raft_consensus.cc:729] T 0caa10cb2bee455eb70a3dae38460c4d P b9d3e9ca17bd4d9babad18338ae019ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d3e9ca17bd4d9babad18338ae019ee, State: Initialized, Role: FOLLOWER
I20260619 18:43:02.252451 30105 consensus_queue.cc:260] T 0caa10cb2bee455eb70a3dae38460c4d P 045f04c1759a4d9bbf38cc06b397ba76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } }
I20260619 18:43:02.252475 30107 consensus_queue.cc:260] T 0caa10cb2bee455eb70a3dae38460c4d P b9d3e9ca17bd4d9babad18338ae019ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } }
I20260619 18:43:02.252717 30105 ts_tablet_manager.cc:1428] T 0caa10cb2bee455eb70a3dae38460c4d P 045f04c1759a4d9bbf38cc06b397ba76: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 18:43:02.252717 30107 ts_tablet_manager.cc:1428] T 0caa10cb2bee455eb70a3dae38460c4d P b9d3e9ca17bd4d9babad18338ae019ee: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 18:43:02.252698 30112 tablet_replica.cc:439] T 0caa10cb2bee455eb70a3dae38460c4d P b9d3e9ca17bd4d9babad18338ae019ee: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } } }
I20260619 18:43:02.252787 30106 raft_consensus.cc:348] T 0caa10cb2bee455eb70a3dae38460c4d P 5ee365d535f64fda9b04ac44dd08738c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } }
I20260619 18:43:02.252878 30106 raft_consensus.cc:374] T 0caa10cb2bee455eb70a3dae38460c4d P 5ee365d535f64fda9b04ac44dd08738c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:43:02.252885 30112 tablet_replica.cc:442] T 0caa10cb2bee455eb70a3dae38460c4d P b9d3e9ca17bd4d9babad18338ae019ee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:43:02.252914 30106 raft_consensus.cc:729] T 0caa10cb2bee455eb70a3dae38460c4d P 5ee365d535f64fda9b04ac44dd08738c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee365d535f64fda9b04ac44dd08738c, State: Initialized, Role: FOLLOWER
I20260619 18:43:02.252925 30129 tablet_replica.cc:439] T 0caa10cb2bee455eb70a3dae38460c4d P 045f04c1759a4d9bbf38cc06b397ba76: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } } }
I20260619 18:43:02.253064 30129 tablet_replica.cc:442] T 0caa10cb2bee455eb70a3dae38460c4d P 045f04c1759a4d9bbf38cc06b397ba76: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:43:02.253026 30106 consensus_queue.cc:260] T 0caa10cb2bee455eb70a3dae38460c4d P 5ee365d535f64fda9b04ac44dd08738c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } }
I20260619 18:43:02.253254 30106 ts_tablet_manager.cc:1428] T 0caa10cb2bee455eb70a3dae38460c4d P 5ee365d535f64fda9b04ac44dd08738c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:43:02.253475 30113 tablet_replica.cc:439] T 0caa10cb2bee455eb70a3dae38460c4d P 5ee365d535f64fda9b04ac44dd08738c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } } }
I20260619 18:43:02.253585 30113 tablet_replica.cc:442] T 0caa10cb2bee455eb70a3dae38460c4d P 5ee365d535f64fda9b04ac44dd08738c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:43:02.533392 30113 raft_consensus.cc:482] T 0caa10cb2bee455eb70a3dae38460c4d P 5ee365d535f64fda9b04ac44dd08738c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:43:02.533552 30113 raft_consensus.cc:504] T 0caa10cb2bee455eb70a3dae38460c4d P 5ee365d535f64fda9b04ac44dd08738c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } }
I20260619 18:43:02.534117 30113 leader_election.cc:288] T 0caa10cb2bee455eb70a3dae38460c4d P 5ee365d535f64fda9b04ac44dd08738c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 045f04c1759a4d9bbf38cc06b397ba76 (127.21.143.65:36541), b9d3e9ca17bd4d9babad18338ae019ee (127.21.143.66:39881)
I20260619 18:43:02.534739 29751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0caa10cb2bee455eb70a3dae38460c4d" candidate_uuid: "5ee365d535f64fda9b04ac44dd08738c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "045f04c1759a4d9bbf38cc06b397ba76" is_pre_election: true
I20260619 18:43:02.534739 29891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0caa10cb2bee455eb70a3dae38460c4d" candidate_uuid: "5ee365d535f64fda9b04ac44dd08738c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" is_pre_election: true
I20260619 18:43:02.534898 29891 raft_consensus.cc:2454] T 0caa10cb2bee455eb70a3dae38460c4d P b9d3e9ca17bd4d9babad18338ae019ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ee365d535f64fda9b04ac44dd08738c in term 0.
I20260619 18:43:02.534898 29751 raft_consensus.cc:2454] T 0caa10cb2bee455eb70a3dae38460c4d P 045f04c1759a4d9bbf38cc06b397ba76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ee365d535f64fda9b04ac44dd08738c in term 0.
I20260619 18:43:02.535209 29964 leader_election.cc:302] T 0caa10cb2bee455eb70a3dae38460c4d P 5ee365d535f64fda9b04ac44dd08738c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ee365d535f64fda9b04ac44dd08738c, b9d3e9ca17bd4d9babad18338ae019ee; no voters:
I20260619 18:43:02.535394 30113 raft_consensus.cc:2781] T 0caa10cb2bee455eb70a3dae38460c4d P 5ee365d535f64fda9b04ac44dd08738c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:43:02.535454 30113 raft_consensus.cc:482] T 0caa10cb2bee455eb70a3dae38460c4d P 5ee365d535f64fda9b04ac44dd08738c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:43:02.535494 30113 raft_consensus.cc:3037] T 0caa10cb2bee455eb70a3dae38460c4d P 5ee365d535f64fda9b04ac44dd08738c [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:43:02.536418 30113 raft_consensus.cc:504] T 0caa10cb2bee455eb70a3dae38460c4d P 5ee365d535f64fda9b04ac44dd08738c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } }
I20260619 18:43:02.536846 30113 leader_election.cc:288] T 0caa10cb2bee455eb70a3dae38460c4d P 5ee365d535f64fda9b04ac44dd08738c [CANDIDATE]: Term 1 election: Requested vote from peers 045f04c1759a4d9bbf38cc06b397ba76 (127.21.143.65:36541), b9d3e9ca17bd4d9babad18338ae019ee (127.21.143.66:39881)
I20260619 18:43:02.537218 29751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0caa10cb2bee455eb70a3dae38460c4d" candidate_uuid: "5ee365d535f64fda9b04ac44dd08738c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "045f04c1759a4d9bbf38cc06b397ba76"
I20260619 18:43:02.537350 29751 raft_consensus.cc:3037] T 0caa10cb2bee455eb70a3dae38460c4d P 045f04c1759a4d9bbf38cc06b397ba76 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:43:02.537510 29891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0caa10cb2bee455eb70a3dae38460c4d" candidate_uuid: "5ee365d535f64fda9b04ac44dd08738c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d3e9ca17bd4d9babad18338ae019ee"
I20260619 18:43:02.537647 29891 raft_consensus.cc:3037] T 0caa10cb2bee455eb70a3dae38460c4d P b9d3e9ca17bd4d9babad18338ae019ee [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:43:02.538317 29751 raft_consensus.cc:2454] T 0caa10cb2bee455eb70a3dae38460c4d P 045f04c1759a4d9bbf38cc06b397ba76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ee365d535f64fda9b04ac44dd08738c in term 1.
I20260619 18:43:02.538605 29964 leader_election.cc:302] T 0caa10cb2bee455eb70a3dae38460c4d P 5ee365d535f64fda9b04ac44dd08738c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 045f04c1759a4d9bbf38cc06b397ba76, 5ee365d535f64fda9b04ac44dd08738c; no voters:
I20260619 18:43:02.538605 29891 raft_consensus.cc:2454] T 0caa10cb2bee455eb70a3dae38460c4d P b9d3e9ca17bd4d9babad18338ae019ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ee365d535f64fda9b04ac44dd08738c in term 1.
I20260619 18:43:02.538774 30113 raft_consensus.cc:2781] T 0caa10cb2bee455eb70a3dae38460c4d P 5ee365d535f64fda9b04ac44dd08738c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:43:02.538868 30113 raft_consensus.cc:686] T 0caa10cb2bee455eb70a3dae38460c4d P 5ee365d535f64fda9b04ac44dd08738c [term 1 LEADER]: Becoming Leader. State: Replica: 5ee365d535f64fda9b04ac44dd08738c, State: Running, Role: LEADER
I20260619 18:43:02.539032 30113 consensus_queue.cc:237] T 0caa10cb2bee455eb70a3dae38460c4d P 5ee365d535f64fda9b04ac44dd08738c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } }
I20260619 18:43:02.539750 30139 tablet_replica.cc:439] T 0caa10cb2bee455eb70a3dae38460c4d P 5ee365d535f64fda9b04ac44dd08738c: TxnStatusTablet state changed. Reason: New leader 5ee365d535f64fda9b04ac44dd08738c. Latest consensus state: current_term: 1 leader_uuid: "5ee365d535f64fda9b04ac44dd08738c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } } }
I20260619 18:43:02.539848 30139 tablet_replica.cc:442] T 0caa10cb2bee455eb70a3dae38460c4d P 5ee365d535f64fda9b04ac44dd08738c: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:43:02.540087 30152 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 18:43:02.540254 30152 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 18:43:02.540722 29593 catalog_manager.cc:5495] T 0caa10cb2bee455eb70a3dae38460c4d P 5ee365d535f64fda9b04ac44dd08738c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ee365d535f64fda9b04ac44dd08738c (127.21.143.67). New cstate: current_term: 1 leader_uuid: "5ee365d535f64fda9b04ac44dd08738c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } health_report { overall_health: HEALTHY } } }
I20260619 18:43:02.613596 30139 consensus_queue.cc:1035] T 2921c09567e64765ab5576f2f192216f P 5ee365d535f64fda9b04ac44dd08738c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:43:02.618428 30113 consensus_queue.cc:1035] T 2921c09567e64765ab5576f2f192216f P 5ee365d535f64fda9b04ac44dd08738c [LEADER]: Connected to new peer: Peer: permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:43:02.624209 30146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:02.617615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:38273 (local address 127.0.0.1:43122)
0619 18:43:02.619234 (+ 1619us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:02.619250 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:02.619381 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:02.620279 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:02.620282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:02.620291 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:43:02.620405 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:02.620411 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:02.621549 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:02.621553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:02.622240 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:02.622250 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:02.622878 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:02.622882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:02.622978 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:02.623041 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:43:02.623960 (+ 919us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:43:02.623974 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 18:43:02.624037 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 18:43:02.624107 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1511}
I20260619 18:43:02.624266 30158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:02.618624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43122 (local address 127.21.143.67:38273)
0619 18:43:02.618901 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:02.618911 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:02.620002 (+ 1091us) server_negotiation.cc:407] Connection header received
0619 18:43:02.620070 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:02.620073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:02.620130 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:02.620513 (+ 383us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:43:02.620604 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:02.621429 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:02.622364 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:02.622742 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:02.622843 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:02.623321 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:43:02.623863 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:43:02.623986 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:02.624070 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 18:43:02.624125 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":108,"threads_started":1}
I20260619 18:43:02.626906 29751 raft_consensus.cc:1261] T 0caa10cb2bee455eb70a3dae38460c4d P 045f04c1759a4d9bbf38cc06b397ba76 [term 1 FOLLOWER]: Refusing update from remote peer 5ee365d535f64fda9b04ac44dd08738c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:43:02.626901 29891 raft_consensus.cc:1261] T 0caa10cb2bee455eb70a3dae38460c4d P b9d3e9ca17bd4d9babad18338ae019ee [term 1 FOLLOWER]: Refusing update from remote peer 5ee365d535f64fda9b04ac44dd08738c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:43:02.627344 30139 consensus_queue.cc:1035] T 0caa10cb2bee455eb70a3dae38460c4d P 5ee365d535f64fda9b04ac44dd08738c [LEADER]: Connected to new peer: Peer: permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:43:02.627556 30113 consensus_queue.cc:1035] T 0caa10cb2bee455eb70a3dae38460c4d P 5ee365d535f64fda9b04ac44dd08738c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:43:02.628616 30129 tablet_replica.cc:439] T 0caa10cb2bee455eb70a3dae38460c4d P 045f04c1759a4d9bbf38cc06b397ba76: TxnStatusTablet state changed. Reason: New leader 5ee365d535f64fda9b04ac44dd08738c. Latest consensus state: current_term: 1 leader_uuid: "5ee365d535f64fda9b04ac44dd08738c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } } }
I20260619 18:43:02.628738 30129 tablet_replica.cc:442] T 0caa10cb2bee455eb70a3dae38460c4d P 045f04c1759a4d9bbf38cc06b397ba76: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:43:02.628818 30112 tablet_replica.cc:439] T 0caa10cb2bee455eb70a3dae38460c4d P b9d3e9ca17bd4d9babad18338ae019ee: TxnStatusTablet state changed. Reason: New leader 5ee365d535f64fda9b04ac44dd08738c. Latest consensus state: current_term: 1 leader_uuid: "5ee365d535f64fda9b04ac44dd08738c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } } }
I20260619 18:43:02.628921 30112 tablet_replica.cc:442] T 0caa10cb2bee455eb70a3dae38460c4d P b9d3e9ca17bd4d9babad18338ae019ee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:43:02.630458 30139 tablet_replica.cc:439] T 0caa10cb2bee455eb70a3dae38460c4d P 5ee365d535f64fda9b04ac44dd08738c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5ee365d535f64fda9b04ac44dd08738c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } } }
I20260619 18:43:02.630574 30139 tablet_replica.cc:442] T 0caa10cb2bee455eb70a3dae38460c4d P 5ee365d535f64fda9b04ac44dd08738c: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:43:02.631183 30112 tablet_replica.cc:439] T 0caa10cb2bee455eb70a3dae38460c4d P b9d3e9ca17bd4d9babad18338ae019ee: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5ee365d535f64fda9b04ac44dd08738c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } } }
I20260619 18:43:02.631482 30156 mvcc.cc:205] Tried to move back new op lower bound from 7298640210436009984 to 7298640210081189888. Current Snapshot: MvccSnapshot[applied={T|T < 7298640210436009984}]
I20260619 18:43:02.631282 30112 tablet_replica.cc:442] T 0caa10cb2bee455eb70a3dae38460c4d P b9d3e9ca17bd4d9babad18338ae019ee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:43:02.632509 30113 tablet_replica.cc:439] T 0caa10cb2bee455eb70a3dae38460c4d P 5ee365d535f64fda9b04ac44dd08738c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5ee365d535f64fda9b04ac44dd08738c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } } }
I20260619 18:43:02.632627 30113 tablet_replica.cc:442] T 0caa10cb2bee455eb70a3dae38460c4d P 5ee365d535f64fda9b04ac44dd08738c: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:43:02.633352 30129 tablet_replica.cc:439] T 0caa10cb2bee455eb70a3dae38460c4d P 045f04c1759a4d9bbf38cc06b397ba76: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5ee365d535f64fda9b04ac44dd08738c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } } }
I20260619 18:43:02.633471 30129 tablet_replica.cc:442] T 0caa10cb2bee455eb70a3dae38460c4d P 045f04c1759a4d9bbf38cc06b397ba76: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:43:02.643280 30158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:02.640106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43138 (local address 127.21.143.67:38273)
0619 18:43:02.640240 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:02.640244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:02.640259 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:43:02.640300 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:02.640303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:02.640346 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:02.640434 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:43:02.640626 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:02.641393 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:02.642227 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:02.642573 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:02.642674 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:02.642868 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:43:02.642952 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:43:02.643017 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:02.643096 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 18:43:02.643149 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:43:02.653518 30158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:02.649856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:43187 (local address 127.21.143.67:38273)
0619 18:43:02.649967 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:02.649970 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:02.650245 (+ 275us) server_negotiation.cc:407] Connection header received
0619 18:43:02.650373 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:02.650375 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:02.650413 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:02.650466 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:43:02.650757 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:02.651219 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:02.652526 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:02.652733 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:02.652806 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:02.653146 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:43:02.653206 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:43:02.653251 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:02.653362 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 18:43:02.653401 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 18:43:02.653693 30171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:02.649797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:38273 (local address 127.21.143.67:43187)
0619 18:43:02.650165 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:02.650180 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:02.650276 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:02.650496 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:02.650502 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:02.650509 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:43:02.650662 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:02.650668 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:02.651351 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:02.651354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:02.652399 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:02.652413 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:02.652854 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:02.652860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:02.652960 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:02.653006 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:43:02.653277 (+ 271us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:43:02.653283 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 18:43:02.653348 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 18:43:02.653423 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":116,"threads_started":1}
I20260619 18:43:02.696331 29594 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:43:02.696507 29594 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:43:02.698649 29594 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3f24326df2f48d7bff275e80e8cbcfe: Sending DeleteTablet for 3 replicas of tablet 2921c09567e64765ab5576f2f192216f
I20260619 18:43:02.699245 29731 tablet_service.cc:1514] Processing DeleteTablet for tablet 2921c09567e64765ab5576f2f192216f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:43:02 UTC) from {username='oryx', principal='oryx/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:45454
I20260619 18:43:02.699342 29871 tablet_service.cc:1514] Processing DeleteTablet for tablet 2921c09567e64765ab5576f2f192216f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:43:02 UTC) from {username='oryx', principal='oryx/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:55776
I20260619 18:43:02.699539 30010 tablet_service.cc:1514] Processing DeleteTablet for tablet 2921c09567e64765ab5576f2f192216f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:43:02 UTC) from {username='oryx', principal='oryx/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:43112
I20260619 18:43:02.699678 30175 tablet_replica.cc:330] T 2921c09567e64765ab5576f2f192216f P b9d3e9ca17bd4d9babad18338ae019ee: stopping tablet replica
I20260619 18:43:02.699807 30175 raft_consensus.cc:2229] T 2921c09567e64765ab5576f2f192216f P b9d3e9ca17bd4d9babad18338ae019ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:43:02.699939 30175 raft_consensus.cc:2258] T 2921c09567e64765ab5576f2f192216f P b9d3e9ca17bd4d9babad18338ae019ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:43:02.700022 30176 tablet_replica.cc:330] T 2921c09567e64765ab5576f2f192216f P 5ee365d535f64fda9b04ac44dd08738c: stopping tablet replica
I20260619 18:43:02.700150 30176 raft_consensus.cc:2229] T 2921c09567e64765ab5576f2f192216f P 5ee365d535f64fda9b04ac44dd08738c [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:43:02.700455 30176 raft_consensus.cc:2258] T 2921c09567e64765ab5576f2f192216f P 5ee365d535f64fda9b04ac44dd08738c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:43:02.700631 30175 ts_tablet_manager.cc:1905] T 2921c09567e64765ab5576f2f192216f P b9d3e9ca17bd4d9babad18338ae019ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:43:02.701159 30176 ts_tablet_manager.cc:1905] T 2921c09567e64765ab5576f2f192216f P 5ee365d535f64fda9b04ac44dd08738c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:43:02.703632 30175 ts_tablet_manager.cc:1918] T 2921c09567e64765ab5576f2f192216f P b9d3e9ca17bd4d9babad18338ae019ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:43:02.703699 30175 log.cc:1196] T 2921c09567e64765ab5576f2f192216f P b9d3e9ca17bd4d9babad18338ae019ee: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-1/wal/wals/2921c09567e64765ab5576f2f192216f
I20260619 18:43:02.703951 30175 ts_tablet_manager.cc:1939] T 2921c09567e64765ab5576f2f192216f P b9d3e9ca17bd4d9babad18338ae019ee: Deleting consensus metadata
I20260619 18:43:02.703954 30174 tablet_replica.cc:330] T 2921c09567e64765ab5576f2f192216f P 045f04c1759a4d9bbf38cc06b397ba76: stopping tablet replica
I20260619 18:43:02.704187 30174 raft_consensus.cc:2229] T 2921c09567e64765ab5576f2f192216f P 045f04c1759a4d9bbf38cc06b397ba76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:43:02.704380 30174 raft_consensus.cc:2258] T 2921c09567e64765ab5576f2f192216f P 045f04c1759a4d9bbf38cc06b397ba76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:43:02.704509 30176 ts_tablet_manager.cc:1918] T 2921c09567e64765ab5576f2f192216f P 5ee365d535f64fda9b04ac44dd08738c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:43:02.704579 30176 log.cc:1196] T 2921c09567e64765ab5576f2f192216f P 5ee365d535f64fda9b04ac44dd08738c: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-2/wal/wals/2921c09567e64765ab5576f2f192216f
I20260619 18:43:02.704591 29580 catalog_manager.cc:4841] TS b9d3e9ca17bd4d9babad18338ae019ee (127.21.143.66:39881): tablet 2921c09567e64765ab5576f2f192216f (table test-table [id=5edb2059fe624ab396438019b4e4fe57]) successfully deleted
I20260619 18:43:02.704839 30176 ts_tablet_manager.cc:1939] T 2921c09567e64765ab5576f2f192216f P 5ee365d535f64fda9b04ac44dd08738c: Deleting consensus metadata
I20260619 18:43:02.705351 30174 ts_tablet_manager.cc:1905] T 2921c09567e64765ab5576f2f192216f P 045f04c1759a4d9bbf38cc06b397ba76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:43:02.705541 29578 catalog_manager.cc:4841] TS 5ee365d535f64fda9b04ac44dd08738c (127.21.143.67:38273): tablet 2921c09567e64765ab5576f2f192216f (table test-table [id=5edb2059fe624ab396438019b4e4fe57]) successfully deleted
I20260619 18:43:02.709007 30174 ts_tablet_manager.cc:1918] T 2921c09567e64765ab5576f2f192216f P 045f04c1759a4d9bbf38cc06b397ba76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:43:02.709158 30174 log.cc:1196] T 2921c09567e64765ab5576f2f192216f P 045f04c1759a4d9bbf38cc06b397ba76: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-0/wal/wals/2921c09567e64765ab5576f2f192216f
I20260619 18:43:02.709491 30174 ts_tablet_manager.cc:1939] T 2921c09567e64765ab5576f2f192216f P 045f04c1759a4d9bbf38cc06b397ba76: Deleting consensus metadata
I20260619 18:43:02.710238 29580 catalog_manager.cc:4841] TS 045f04c1759a4d9bbf38cc06b397ba76 (127.21.143.65:36541): tablet 2921c09567e64765ab5576f2f192216f (table test-table [id=5edb2059fe624ab396438019b4e4fe57]) successfully deleted
Jun 19 18:43:02 dist-test-slave-mld5 krb5kdc[29549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 18:43:02 dist-test-slave-mld5 krb5kdc[29549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:02 dist-test-slave-mld5 krb5kdc[29549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 18:43:02.729566 30147 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:02.722633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46522 (local address 127.21.143.126:33147)
0619 18:43:02.722789 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:02.722793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:02.722981 (+ 188us) server_negotiation.cc:407] Connection header received
0619 18:43:02.723118 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:02.723120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:02.723166 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:02.723235 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:02.724146 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:02.724677 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:02.725917 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:02.726153 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:02.726303 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:02.729443 (+ 3140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46522: BlockingRecv error: recv got EOF from 127.0.0.1:46522 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 18:43:02 dist-test-slave-mld5 krb5kdc[29549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM
I20260619 18:43:02.739861 30147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:02.731017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46536 (local address 127.21.143.126:33147)
0619 18:43:02.731145 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:02.731151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:02.731839 (+ 688us) server_negotiation.cc:407] Connection header received
0619 18:43:02.731884 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:02.731887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:02.731940 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:02.732022 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:02.732579 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:02.733356 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:02.734135 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:02.734471 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:02.734569 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:02.736949 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:02.736952 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:02.736954 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:02.736974 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:02.738312 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:02.738745 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:02.738748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:02.738749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:02.738817 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:02.739075 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:02.739077 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:02.739078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:02.739230 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:02.739401 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:02.739605 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 18:43:02.739696 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 18:43:02.742290 29594 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:43:02.742728 29594 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:43:02.747133 29731 tablet_service.cc:1467] Processing CreateTablet for tablet ec3d5c378a774f09b84dcecc3760bfa0 (DEFAULT_TABLE table=test-table [id=b0405faf0c114c96a1742a4be32235e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:43:02.747223 29871 tablet_service.cc:1467] Processing CreateTablet for tablet ec3d5c378a774f09b84dcecc3760bfa0 (DEFAULT_TABLE table=test-table [id=b0405faf0c114c96a1742a4be32235e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:43:02.747390 30010 tablet_service.cc:1467] Processing CreateTablet for tablet ec3d5c378a774f09b84dcecc3760bfa0 (DEFAULT_TABLE table=test-table [id=b0405faf0c114c96a1742a4be32235e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:43:02.747460 29731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec3d5c378a774f09b84dcecc3760bfa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:43:02.747540 29871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec3d5c378a774f09b84dcecc3760bfa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:43:02.747633 30010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec3d5c378a774f09b84dcecc3760bfa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:43:02.749841 30105 tablet_bootstrap.cc:492] T ec3d5c378a774f09b84dcecc3760bfa0 P 045f04c1759a4d9bbf38cc06b397ba76: Bootstrap starting.
I20260619 18:43:02.749887 30107 tablet_bootstrap.cc:492] T ec3d5c378a774f09b84dcecc3760bfa0 P b9d3e9ca17bd4d9babad18338ae019ee: Bootstrap starting.
I20260619 18:43:02.750665 30107 tablet_bootstrap.cc:654] T ec3d5c378a774f09b84dcecc3760bfa0 P b9d3e9ca17bd4d9babad18338ae019ee: Neither blocks nor log segments found. Creating new log.
I20260619 18:43:02.750699 30105 tablet_bootstrap.cc:654] T ec3d5c378a774f09b84dcecc3760bfa0 P 045f04c1759a4d9bbf38cc06b397ba76: Neither blocks nor log segments found. Creating new log.
I20260619 18:43:02.751179 30106 tablet_bootstrap.cc:492] T ec3d5c378a774f09b84dcecc3760bfa0 P 5ee365d535f64fda9b04ac44dd08738c: Bootstrap starting.
I20260619 18:43:02.752192 30106 tablet_bootstrap.cc:654] T ec3d5c378a774f09b84dcecc3760bfa0 P 5ee365d535f64fda9b04ac44dd08738c: Neither blocks nor log segments found. Creating new log.
I20260619 18:43:02.753046 30107 tablet_bootstrap.cc:492] T ec3d5c378a774f09b84dcecc3760bfa0 P b9d3e9ca17bd4d9babad18338ae019ee: No bootstrap required, opened a new log
I20260619 18:43:02.753146 30107 ts_tablet_manager.cc:1397] T ec3d5c378a774f09b84dcecc3760bfa0 P b9d3e9ca17bd4d9babad18338ae019ee: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 18:43:02.753238 30106 tablet_bootstrap.cc:492] T ec3d5c378a774f09b84dcecc3760bfa0 P 5ee365d535f64fda9b04ac44dd08738c: No bootstrap required, opened a new log
I20260619 18:43:02.753322 30106 ts_tablet_manager.cc:1397] T ec3d5c378a774f09b84dcecc3760bfa0 P 5ee365d535f64fda9b04ac44dd08738c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 18:43:02.753587 30107 raft_consensus.cc:348] T ec3d5c378a774f09b84dcecc3760bfa0 P b9d3e9ca17bd4d9babad18338ae019ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } }
I20260619 18:43:02.753703 30107 raft_consensus.cc:374] T ec3d5c378a774f09b84dcecc3760bfa0 P b9d3e9ca17bd4d9babad18338ae019ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:43:02.753684 30106 raft_consensus.cc:348] T ec3d5c378a774f09b84dcecc3760bfa0 P 5ee365d535f64fda9b04ac44dd08738c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } }
I20260619 18:43:02.753742 30107 raft_consensus.cc:729] T ec3d5c378a774f09b84dcecc3760bfa0 P b9d3e9ca17bd4d9babad18338ae019ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d3e9ca17bd4d9babad18338ae019ee, State: Initialized, Role: FOLLOWER
I20260619 18:43:02.753788 30106 raft_consensus.cc:374] T ec3d5c378a774f09b84dcecc3760bfa0 P 5ee365d535f64fda9b04ac44dd08738c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:43:02.753818 30106 raft_consensus.cc:729] T ec3d5c378a774f09b84dcecc3760bfa0 P 5ee365d535f64fda9b04ac44dd08738c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee365d535f64fda9b04ac44dd08738c, State: Initialized, Role: FOLLOWER
I20260619 18:43:02.753870 30107 consensus_queue.cc:260] T ec3d5c378a774f09b84dcecc3760bfa0 P b9d3e9ca17bd4d9babad18338ae019ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } }
I20260619 18:43:02.753925 30106 consensus_queue.cc:260] T ec3d5c378a774f09b84dcecc3760bfa0 P 5ee365d535f64fda9b04ac44dd08738c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } }
I20260619 18:43:02.754156 30107 ts_tablet_manager.cc:1428] T ec3d5c378a774f09b84dcecc3760bfa0 P b9d3e9ca17bd4d9babad18338ae019ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:43:02.754168 30106 ts_tablet_manager.cc:1428] T ec3d5c378a774f09b84dcecc3760bfa0 P 5ee365d535f64fda9b04ac44dd08738c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 18:43:02.755918 30105 tablet_bootstrap.cc:492] T ec3d5c378a774f09b84dcecc3760bfa0 P 045f04c1759a4d9bbf38cc06b397ba76: No bootstrap required, opened a new log
I20260619 18:43:02.756013 30105 ts_tablet_manager.cc:1397] T ec3d5c378a774f09b84dcecc3760bfa0 P 045f04c1759a4d9bbf38cc06b397ba76: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260619 18:43:02.756388 30105 raft_consensus.cc:348] T ec3d5c378a774f09b84dcecc3760bfa0 P 045f04c1759a4d9bbf38cc06b397ba76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } }
I20260619 18:43:02.756503 30105 raft_consensus.cc:374] T ec3d5c378a774f09b84dcecc3760bfa0 P 045f04c1759a4d9bbf38cc06b397ba76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:43:02.756536 30105 raft_consensus.cc:729] T ec3d5c378a774f09b84dcecc3760bfa0 P 045f04c1759a4d9bbf38cc06b397ba76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 045f04c1759a4d9bbf38cc06b397ba76, State: Initialized, Role: FOLLOWER
I20260619 18:43:02.756655 30105 consensus_queue.cc:260] T ec3d5c378a774f09b84dcecc3760bfa0 P 045f04c1759a4d9bbf38cc06b397ba76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } }
I20260619 18:43:02.756757 30129 raft_consensus.cc:482] T ec3d5c378a774f09b84dcecc3760bfa0 P 045f04c1759a4d9bbf38cc06b397ba76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:43:02.756862 30129 raft_consensus.cc:504] T ec3d5c378a774f09b84dcecc3760bfa0 P 045f04c1759a4d9bbf38cc06b397ba76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } }
I20260619 18:43:02.757005 30105 ts_tablet_manager.cc:1428] T ec3d5c378a774f09b84dcecc3760bfa0 P 045f04c1759a4d9bbf38cc06b397ba76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:43:02.757249 30129 leader_election.cc:288] T ec3d5c378a774f09b84dcecc3760bfa0 P 045f04c1759a4d9bbf38cc06b397ba76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9d3e9ca17bd4d9babad18338ae019ee (127.21.143.66:39881), 5ee365d535f64fda9b04ac44dd08738c (127.21.143.67:38273)
I20260619 18:43:02.757647 29891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec3d5c378a774f09b84dcecc3760bfa0" candidate_uuid: "045f04c1759a4d9bbf38cc06b397ba76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" is_pre_election: true
I20260619 18:43:02.757779 29891 raft_consensus.cc:2454] T ec3d5c378a774f09b84dcecc3760bfa0 P b9d3e9ca17bd4d9babad18338ae019ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 045f04c1759a4d9bbf38cc06b397ba76 in term 0.
I20260619 18:43:02.758060 30031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec3d5c378a774f09b84dcecc3760bfa0" candidate_uuid: "045f04c1759a4d9bbf38cc06b397ba76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ee365d535f64fda9b04ac44dd08738c" is_pre_election: true
I20260619 18:43:02.758141 29684 leader_election.cc:302] T ec3d5c378a774f09b84dcecc3760bfa0 P 045f04c1759a4d9bbf38cc06b397ba76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 045f04c1759a4d9bbf38cc06b397ba76, b9d3e9ca17bd4d9babad18338ae019ee; no voters:
I20260619 18:43:02.758193 30031 raft_consensus.cc:2454] T ec3d5c378a774f09b84dcecc3760bfa0 P 5ee365d535f64fda9b04ac44dd08738c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 045f04c1759a4d9bbf38cc06b397ba76 in term 0.
I20260619 18:43:02.758388 30129 raft_consensus.cc:2781] T ec3d5c378a774f09b84dcecc3760bfa0 P 045f04c1759a4d9bbf38cc06b397ba76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:43:02.758441 30129 raft_consensus.cc:482] T ec3d5c378a774f09b84dcecc3760bfa0 P 045f04c1759a4d9bbf38cc06b397ba76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:43:02.758479 30129 raft_consensus.cc:3037] T ec3d5c378a774f09b84dcecc3760bfa0 P 045f04c1759a4d9bbf38cc06b397ba76 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:43:02.759651 30129 raft_consensus.cc:504] T ec3d5c378a774f09b84dcecc3760bfa0 P 045f04c1759a4d9bbf38cc06b397ba76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } }
I20260619 18:43:02.760010 30129 leader_election.cc:288] T ec3d5c378a774f09b84dcecc3760bfa0 P 045f04c1759a4d9bbf38cc06b397ba76 [CANDIDATE]: Term 1 election: Requested vote from peers b9d3e9ca17bd4d9babad18338ae019ee (127.21.143.66:39881), 5ee365d535f64fda9b04ac44dd08738c (127.21.143.67:38273)
I20260619 18:43:02.760378 29891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec3d5c378a774f09b84dcecc3760bfa0" candidate_uuid: "045f04c1759a4d9bbf38cc06b397ba76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d3e9ca17bd4d9babad18338ae019ee"
I20260619 18:43:02.760442 30031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec3d5c378a774f09b84dcecc3760bfa0" candidate_uuid: "045f04c1759a4d9bbf38cc06b397ba76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ee365d535f64fda9b04ac44dd08738c"
I20260619 18:43:02.760494 29891 raft_consensus.cc:3037] T ec3d5c378a774f09b84dcecc3760bfa0 P b9d3e9ca17bd4d9babad18338ae019ee [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:43:02.760562 30031 raft_consensus.cc:3037] T ec3d5c378a774f09b84dcecc3760bfa0 P 5ee365d535f64fda9b04ac44dd08738c [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:43:02.761482 30031 raft_consensus.cc:2454] T ec3d5c378a774f09b84dcecc3760bfa0 P 5ee365d535f64fda9b04ac44dd08738c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 045f04c1759a4d9bbf38cc06b397ba76 in term 1.
I20260619 18:43:02.761509 29891 raft_consensus.cc:2454] T ec3d5c378a774f09b84dcecc3760bfa0 P b9d3e9ca17bd4d9babad18338ae019ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 045f04c1759a4d9bbf38cc06b397ba76 in term 1.
I20260619 18:43:02.761821 29684 leader_election.cc:302] T ec3d5c378a774f09b84dcecc3760bfa0 P 045f04c1759a4d9bbf38cc06b397ba76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 045f04c1759a4d9bbf38cc06b397ba76, 5ee365d535f64fda9b04ac44dd08738c, b9d3e9ca17bd4d9babad18338ae019ee; no voters:
I20260619 18:43:02.762030 30129 raft_consensus.cc:2781] T ec3d5c378a774f09b84dcecc3760bfa0 P 045f04c1759a4d9bbf38cc06b397ba76 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:43:02.762102 30129 raft_consensus.cc:686] T ec3d5c378a774f09b84dcecc3760bfa0 P 045f04c1759a4d9bbf38cc06b397ba76 [term 1 LEADER]: Becoming Leader. State: Replica: 045f04c1759a4d9bbf38cc06b397ba76, State: Running, Role: LEADER
I20260619 18:43:02.762207 30129 consensus_queue.cc:237] T ec3d5c378a774f09b84dcecc3760bfa0 P 045f04c1759a4d9bbf38cc06b397ba76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } }
I20260619 18:43:02.763604 29594 catalog_manager.cc:5495] T ec3d5c378a774f09b84dcecc3760bfa0 P 045f04c1759a4d9bbf38cc06b397ba76 reported cstate change: term changed from 0 to 1, leader changed from <none> to 045f04c1759a4d9bbf38cc06b397ba76 (127.21.143.65). New cstate: current_term: 1 leader_uuid: "045f04c1759a4d9bbf38cc06b397ba76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } health_report { overall_health: UNKNOWN } } }
I20260619 18:43:02.774212 30195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:02.770490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45478 (local address 127.21.143.65:36541)
0619 18:43:02.770823 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:02.770833 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:02.770848 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:43:02.770895 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:02.770898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:02.770958 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:02.771072 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:43:02.771345 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:02.772187 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:02.773012 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:02.773383 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:02.773499 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:02.773742 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:43:02.773827 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:43:02.773897 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:02.773992 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 18:43:02.774045 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":118,"threads_started":1}
I20260619 18:43:02.776316 30031 raft_consensus.cc:1261] T ec3d5c378a774f09b84dcecc3760bfa0 P 5ee365d535f64fda9b04ac44dd08738c [term 1 FOLLOWER]: Refusing update from remote peer 045f04c1759a4d9bbf38cc06b397ba76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:43:02.776407 29891 raft_consensus.cc:1261] T ec3d5c378a774f09b84dcecc3760bfa0 P b9d3e9ca17bd4d9babad18338ae019ee [term 1 FOLLOWER]: Refusing update from remote peer 045f04c1759a4d9bbf38cc06b397ba76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:43:02.776889 30192 consensus_queue.cc:1035] T ec3d5c378a774f09b84dcecc3760bfa0 P 045f04c1759a4d9bbf38cc06b397ba76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:43:02.777069 30129 consensus_queue.cc:1035] T ec3d5c378a774f09b84dcecc3760bfa0 P 045f04c1759a4d9bbf38cc06b397ba76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:43:02.784900 29594 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:43:02.785044 29594 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:43:02.787154 29594 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3f24326df2f48d7bff275e80e8cbcfe: Sending DeleteTablet for 3 replicas of tablet ec3d5c378a774f09b84dcecc3760bfa0
I20260619 18:43:02.787828 29731 tablet_service.cc:1514] Processing DeleteTablet for tablet ec3d5c378a774f09b84dcecc3760bfa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:43:02 UTC) from {username='oryx', principal='oryx/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:45454
I20260619 18:43:02.787865 30010 tablet_service.cc:1514] Processing DeleteTablet for tablet ec3d5c378a774f09b84dcecc3760bfa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:43:02 UTC) from {username='oryx', principal='oryx/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:43112
I20260619 18:43:02.787853 29871 tablet_service.cc:1514] Processing DeleteTablet for tablet ec3d5c378a774f09b84dcecc3760bfa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:43:02 UTC) from {username='oryx', principal='oryx/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:55776
I20260619 18:43:02.788033 30174 tablet_replica.cc:330] T ec3d5c378a774f09b84dcecc3760bfa0 P 045f04c1759a4d9bbf38cc06b397ba76: stopping tablet replica
I20260619 18:43:02.788080 30175 tablet_replica.cc:330] T ec3d5c378a774f09b84dcecc3760bfa0 P b9d3e9ca17bd4d9babad18338ae019ee: stopping tablet replica
I20260619 18:43:02.788154 30174 raft_consensus.cc:2229] T ec3d5c378a774f09b84dcecc3760bfa0 P 045f04c1759a4d9bbf38cc06b397ba76 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:43:02.788182 30175 raft_consensus.cc:2229] T ec3d5c378a774f09b84dcecc3760bfa0 P b9d3e9ca17bd4d9babad18338ae019ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:43:02.788292 30175 raft_consensus.cc:2258] T ec3d5c378a774f09b84dcecc3760bfa0 P b9d3e9ca17bd4d9babad18338ae019ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:43:02.788407 30174 raft_consensus.cc:2258] T ec3d5c378a774f09b84dcecc3760bfa0 P 045f04c1759a4d9bbf38cc06b397ba76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:43:02.788594 30176 tablet_replica.cc:330] T ec3d5c378a774f09b84dcecc3760bfa0 P 5ee365d535f64fda9b04ac44dd08738c: stopping tablet replica
I20260619 18:43:02.788722 30176 raft_consensus.cc:2229] T ec3d5c378a774f09b84dcecc3760bfa0 P 5ee365d535f64fda9b04ac44dd08738c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:43:02.788847 30176 raft_consensus.cc:2258] T ec3d5c378a774f09b84dcecc3760bfa0 P 5ee365d535f64fda9b04ac44dd08738c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:43:02.789106 30175 ts_tablet_manager.cc:1905] T ec3d5c378a774f09b84dcecc3760bfa0 P b9d3e9ca17bd4d9babad18338ae019ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:43:02.789503 30174 ts_tablet_manager.cc:1905] T ec3d5c378a774f09b84dcecc3760bfa0 P 045f04c1759a4d9bbf38cc06b397ba76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:43:02.791024 30175 ts_tablet_manager.cc:1918] T ec3d5c378a774f09b84dcecc3760bfa0 P b9d3e9ca17bd4d9babad18338ae019ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:43:02.791085 30175 log.cc:1196] T ec3d5c378a774f09b84dcecc3760bfa0 P b9d3e9ca17bd4d9babad18338ae019ee: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-1/wal/wals/ec3d5c378a774f09b84dcecc3760bfa0
I20260619 18:43:02.791286 30175 ts_tablet_manager.cc:1939] T ec3d5c378a774f09b84dcecc3760bfa0 P b9d3e9ca17bd4d9babad18338ae019ee: Deleting consensus metadata
I20260619 18:43:02.791972 30174 ts_tablet_manager.cc:1918] T ec3d5c378a774f09b84dcecc3760bfa0 P 045f04c1759a4d9bbf38cc06b397ba76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:43:02.791915 29594 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 18:43:02.792043 30174 log.cc:1196] T ec3d5c378a774f09b84dcecc3760bfa0 P 045f04c1759a4d9bbf38cc06b397ba76: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-0/wal/wals/ec3d5c378a774f09b84dcecc3760bfa0
I20260619 18:43:02.792066 30176 ts_tablet_manager.cc:1905] T ec3d5c378a774f09b84dcecc3760bfa0 P 5ee365d535f64fda9b04ac44dd08738c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:43:02.792275 30174 ts_tablet_manager.cc:1939] T ec3d5c378a774f09b84dcecc3760bfa0 P 045f04c1759a4d9bbf38cc06b397ba76: Deleting consensus metadata
W20260619 18:43:02.792310 29594 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:43:02.792871 29580 catalog_manager.cc:4841] TS 045f04c1759a4d9bbf38cc06b397ba76 (127.21.143.65:36541): tablet ec3d5c378a774f09b84dcecc3760bfa0 (table test-table [id=b0405faf0c114c96a1742a4be32235e7]) successfully deleted
I20260619 18:43:02.794298 29580 catalog_manager.cc:4841] TS b9d3e9ca17bd4d9babad18338ae019ee (127.21.143.66:39881): tablet ec3d5c378a774f09b84dcecc3760bfa0 (table test-table [id=b0405faf0c114c96a1742a4be32235e7]) successfully deleted
I20260619 18:43:02.794466 30176 ts_tablet_manager.cc:1918] T ec3d5c378a774f09b84dcecc3760bfa0 P 5ee365d535f64fda9b04ac44dd08738c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:43:02.794529 30176 log.cc:1196] T ec3d5c378a774f09b84dcecc3760bfa0 P 5ee365d535f64fda9b04ac44dd08738c: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-2/wal/wals/ec3d5c378a774f09b84dcecc3760bfa0
I20260619 18:43:02.794797 30176 ts_tablet_manager.cc:1939] T ec3d5c378a774f09b84dcecc3760bfa0 P 5ee365d535f64fda9b04ac44dd08738c: Deleting consensus metadata
I20260619 18:43:02.795472 29578 catalog_manager.cc:4841] TS 5ee365d535f64fda9b04ac44dd08738c (127.21.143.67:38273): tablet ec3d5c378a774f09b84dcecc3760bfa0 (table test-table [id=b0405faf0c114c96a1742a4be32235e7]) successfully deleted
I20260619 18:43:02.797614 29731 tablet_service.cc:1467] Processing CreateTablet for tablet bcf9e97e867c4281bad84d81091b295c (DEFAULT_TABLE table=test-table [id=5867ac49e80849ba9dfc49f19421b933]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:43:02.797641 30010 tablet_service.cc:1467] Processing CreateTablet for tablet bcf9e97e867c4281bad84d81091b295c (DEFAULT_TABLE table=test-table [id=5867ac49e80849ba9dfc49f19421b933]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:43:02.797631 29871 tablet_service.cc:1467] Processing CreateTablet for tablet bcf9e97e867c4281bad84d81091b295c (DEFAULT_TABLE table=test-table [id=5867ac49e80849ba9dfc49f19421b933]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:43:02.797881 30010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcf9e97e867c4281bad84d81091b295c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:43:02.797881 29871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcf9e97e867c4281bad84d81091b295c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:43:02.797883 29731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcf9e97e867c4281bad84d81091b295c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:43:02.800248 30107 tablet_bootstrap.cc:492] T bcf9e97e867c4281bad84d81091b295c P b9d3e9ca17bd4d9babad18338ae019ee: Bootstrap starting.
I20260619 18:43:02.800248 30106 tablet_bootstrap.cc:492] T bcf9e97e867c4281bad84d81091b295c P 5ee365d535f64fda9b04ac44dd08738c: Bootstrap starting.
I20260619 18:43:02.800279 30105 tablet_bootstrap.cc:492] T bcf9e97e867c4281bad84d81091b295c P 045f04c1759a4d9bbf38cc06b397ba76: Bootstrap starting.
I20260619 18:43:02.801097 30105 tablet_bootstrap.cc:654] T bcf9e97e867c4281bad84d81091b295c P 045f04c1759a4d9bbf38cc06b397ba76: Neither blocks nor log segments found. Creating new log.
I20260619 18:43:02.801100 30106 tablet_bootstrap.cc:654] T bcf9e97e867c4281bad84d81091b295c P 5ee365d535f64fda9b04ac44dd08738c: Neither blocks nor log segments found. Creating new log.
I20260619 18:43:02.801134 30107 tablet_bootstrap.cc:654] T bcf9e97e867c4281bad84d81091b295c P b9d3e9ca17bd4d9babad18338ae019ee: Neither blocks nor log segments found. Creating new log.
I20260619 18:43:02.802567 30105 tablet_bootstrap.cc:492] T bcf9e97e867c4281bad84d81091b295c P 045f04c1759a4d9bbf38cc06b397ba76: No bootstrap required, opened a new log
I20260619 18:43:02.802655 30105 ts_tablet_manager.cc:1397] T bcf9e97e867c4281bad84d81091b295c P 045f04c1759a4d9bbf38cc06b397ba76: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 18:43:02.803066 30105 raft_consensus.cc:348] T bcf9e97e867c4281bad84d81091b295c P 045f04c1759a4d9bbf38cc06b397ba76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } }
I20260619 18:43:02.803184 30105 raft_consensus.cc:374] T bcf9e97e867c4281bad84d81091b295c P 045f04c1759a4d9bbf38cc06b397ba76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:43:02.803215 30105 raft_consensus.cc:729] T bcf9e97e867c4281bad84d81091b295c P 045f04c1759a4d9bbf38cc06b397ba76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 045f04c1759a4d9bbf38cc06b397ba76, State: Initialized, Role: FOLLOWER
I20260619 18:43:02.803400 30107 tablet_bootstrap.cc:492] T bcf9e97e867c4281bad84d81091b295c P b9d3e9ca17bd4d9babad18338ae019ee: No bootstrap required, opened a new log
I20260619 18:43:02.803400 30106 tablet_bootstrap.cc:492] T bcf9e97e867c4281bad84d81091b295c P 5ee365d535f64fda9b04ac44dd08738c: No bootstrap required, opened a new log
I20260619 18:43:02.803357 30105 consensus_queue.cc:260] T bcf9e97e867c4281bad84d81091b295c P 045f04c1759a4d9bbf38cc06b397ba76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } }
I20260619 18:43:02.803478 30106 ts_tablet_manager.cc:1397] T bcf9e97e867c4281bad84d81091b295c P 5ee365d535f64fda9b04ac44dd08738c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260619 18:43:02.803478 30107 ts_tablet_manager.cc:1397] T bcf9e97e867c4281bad84d81091b295c P b9d3e9ca17bd4d9babad18338ae019ee: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 18:43:02.803637 30105 ts_tablet_manager.cc:1428] T bcf9e97e867c4281bad84d81091b295c P 045f04c1759a4d9bbf38cc06b397ba76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:43:02.803882 30106 raft_consensus.cc:348] T bcf9e97e867c4281bad84d81091b295c P 5ee365d535f64fda9b04ac44dd08738c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } }
I20260619 18:43:02.803987 30106 raft_consensus.cc:374] T bcf9e97e867c4281bad84d81091b295c P 5ee365d535f64fda9b04ac44dd08738c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:43:02.803927 30107 raft_consensus.cc:348] T bcf9e97e867c4281bad84d81091b295c P b9d3e9ca17bd4d9babad18338ae019ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } }
I20260619 18:43:02.804018 30106 raft_consensus.cc:729] T bcf9e97e867c4281bad84d81091b295c P 5ee365d535f64fda9b04ac44dd08738c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee365d535f64fda9b04ac44dd08738c, State: Initialized, Role: FOLLOWER
I20260619 18:43:02.804028 30107 raft_consensus.cc:374] T bcf9e97e867c4281bad84d81091b295c P b9d3e9ca17bd4d9babad18338ae019ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:43:02.804064 30107 raft_consensus.cc:729] T bcf9e97e867c4281bad84d81091b295c P b9d3e9ca17bd4d9babad18338ae019ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d3e9ca17bd4d9babad18338ae019ee, State: Initialized, Role: FOLLOWER
I20260619 18:43:02.804147 30106 consensus_queue.cc:260] T bcf9e97e867c4281bad84d81091b295c P 5ee365d535f64fda9b04ac44dd08738c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } }
I20260619 18:43:02.804198 30107 consensus_queue.cc:260] T bcf9e97e867c4281bad84d81091b295c P b9d3e9ca17bd4d9babad18338ae019ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } }
I20260619 18:43:02.804394 30106 ts_tablet_manager.cc:1428] T bcf9e97e867c4281bad84d81091b295c P 5ee365d535f64fda9b04ac44dd08738c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 18:43:02.804455 30107 ts_tablet_manager.cc:1428] T bcf9e97e867c4281bad84d81091b295c P b9d3e9ca17bd4d9babad18338ae019ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:43:02.871181 30198 raft_consensus.cc:482] T bcf9e97e867c4281bad84d81091b295c P 045f04c1759a4d9bbf38cc06b397ba76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:43:02.871377 30198 raft_consensus.cc:504] T bcf9e97e867c4281bad84d81091b295c P 045f04c1759a4d9bbf38cc06b397ba76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } }
I20260619 18:43:02.871871 30198 leader_election.cc:288] T bcf9e97e867c4281bad84d81091b295c P 045f04c1759a4d9bbf38cc06b397ba76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9d3e9ca17bd4d9babad18338ae019ee (127.21.143.66:39881), 5ee365d535f64fda9b04ac44dd08738c (127.21.143.67:38273)
I20260619 18:43:02.872313 29891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf9e97e867c4281bad84d81091b295c" candidate_uuid: "045f04c1759a4d9bbf38cc06b397ba76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" is_pre_election: true
I20260619 18:43:02.872315 30031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf9e97e867c4281bad84d81091b295c" candidate_uuid: "045f04c1759a4d9bbf38cc06b397ba76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ee365d535f64fda9b04ac44dd08738c" is_pre_election: true
I20260619 18:43:02.872462 30031 raft_consensus.cc:2454] T bcf9e97e867c4281bad84d81091b295c P 5ee365d535f64fda9b04ac44dd08738c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 045f04c1759a4d9bbf38cc06b397ba76 in term 0.
I20260619 18:43:02.872469 29891 raft_consensus.cc:2454] T bcf9e97e867c4281bad84d81091b295c P b9d3e9ca17bd4d9babad18338ae019ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 045f04c1759a4d9bbf38cc06b397ba76 in term 0.
I20260619 18:43:02.872742 29682 leader_election.cc:302] T bcf9e97e867c4281bad84d81091b295c P 045f04c1759a4d9bbf38cc06b397ba76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 045f04c1759a4d9bbf38cc06b397ba76, 5ee365d535f64fda9b04ac44dd08738c; no voters:
I20260619 18:43:02.872918 30198 raft_consensus.cc:2781] T bcf9e97e867c4281bad84d81091b295c P 045f04c1759a4d9bbf38cc06b397ba76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:43:02.872980 30198 raft_consensus.cc:482] T bcf9e97e867c4281bad84d81091b295c P 045f04c1759a4d9bbf38cc06b397ba76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:43:02.873040 30198 raft_consensus.cc:3037] T bcf9e97e867c4281bad84d81091b295c P 045f04c1759a4d9bbf38cc06b397ba76 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:43:02.874017 30198 raft_consensus.cc:504] T bcf9e97e867c4281bad84d81091b295c P 045f04c1759a4d9bbf38cc06b397ba76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } }
I20260619 18:43:02.874377 30198 leader_election.cc:288] T bcf9e97e867c4281bad84d81091b295c P 045f04c1759a4d9bbf38cc06b397ba76 [CANDIDATE]: Term 1 election: Requested vote from peers b9d3e9ca17bd4d9babad18338ae019ee (127.21.143.66:39881), 5ee365d535f64fda9b04ac44dd08738c (127.21.143.67:38273)
I20260619 18:43:02.874688 29891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf9e97e867c4281bad84d81091b295c" candidate_uuid: "045f04c1759a4d9bbf38cc06b397ba76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d3e9ca17bd4d9babad18338ae019ee"
I20260619 18:43:02.874701 30031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf9e97e867c4281bad84d81091b295c" candidate_uuid: "045f04c1759a4d9bbf38cc06b397ba76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ee365d535f64fda9b04ac44dd08738c"
I20260619 18:43:02.874791 29891 raft_consensus.cc:3037] T bcf9e97e867c4281bad84d81091b295c P b9d3e9ca17bd4d9babad18338ae019ee [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:43:02.874845 30031 raft_consensus.cc:3037] T bcf9e97e867c4281bad84d81091b295c P 5ee365d535f64fda9b04ac44dd08738c [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:43:02.875803 29891 raft_consensus.cc:2454] T bcf9e97e867c4281bad84d81091b295c P b9d3e9ca17bd4d9babad18338ae019ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 045f04c1759a4d9bbf38cc06b397ba76 in term 1.
I20260619 18:43:02.875967 30031 raft_consensus.cc:2454] T bcf9e97e867c4281bad84d81091b295c P 5ee365d535f64fda9b04ac44dd08738c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 045f04c1759a4d9bbf38cc06b397ba76 in term 1.
I20260619 18:43:02.876132 29684 leader_election.cc:302] T bcf9e97e867c4281bad84d81091b295c P 045f04c1759a4d9bbf38cc06b397ba76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 045f04c1759a4d9bbf38cc06b397ba76, b9d3e9ca17bd4d9babad18338ae019ee; no voters:
I20260619 18:43:02.876325 30198 raft_consensus.cc:2781] T bcf9e97e867c4281bad84d81091b295c P 045f04c1759a4d9bbf38cc06b397ba76 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:43:02.876417 30198 raft_consensus.cc:686] T bcf9e97e867c4281bad84d81091b295c P 045f04c1759a4d9bbf38cc06b397ba76 [term 1 LEADER]: Becoming Leader. State: Replica: 045f04c1759a4d9bbf38cc06b397ba76, State: Running, Role: LEADER
I20260619 18:43:02.876595 30198 consensus_queue.cc:237] T bcf9e97e867c4281bad84d81091b295c P 045f04c1759a4d9bbf38cc06b397ba76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } }
I20260619 18:43:02.878086 29592 catalog_manager.cc:5495] T bcf9e97e867c4281bad84d81091b295c P 045f04c1759a4d9bbf38cc06b397ba76 reported cstate change: term changed from 0 to 1, leader changed from <none> to 045f04c1759a4d9bbf38cc06b397ba76 (127.21.143.65). New cstate: current_term: 1 leader_uuid: "045f04c1759a4d9bbf38cc06b397ba76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f04c1759a4d9bbf38cc06b397ba76" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 } health_report { overall_health: UNKNOWN } } }
I20260619 18:43:02.898041 30158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:02.895050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:40243 (local address 127.21.143.67:38273)
0619 18:43:02.895200 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:02.895204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:02.895289 (+ 85us) server_negotiation.cc:407] Connection header received
0619 18:43:02.895449 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:02.895451 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:02.895491 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:02.895555 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:43:02.895753 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:02.896224 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:02.897131 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:02.897395 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:02.897477 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:02.897709 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:43:02.897772 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:43:02.897823 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:02.897903 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 18:43:02.897946 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 18:43:02.898077 30205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:02.894926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:38273 (local address 127.21.143.65:40243)
0619 18:43:02.895217 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:02.895230 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:02.895359 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:02.895578 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:02.895580 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:02.895590 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:43:02.895673 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:02.895677 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:02.896304 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:02.896306 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:02.896954 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:02.896961 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:02.897498 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:02.897500 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:02.897563 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:02.897602 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:43:02.897848 (+ 246us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:43:02.897852 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 18:43:02.897896 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 18:43:02.897964 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":120,"threads_started":1}
I20260619 18:43:02.904207 30031 raft_consensus.cc:1261] T bcf9e97e867c4281bad84d81091b295c P 5ee365d535f64fda9b04ac44dd08738c [term 1 FOLLOWER]: Refusing update from remote peer 045f04c1759a4d9bbf38cc06b397ba76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:43:02.904294 29891 raft_consensus.cc:1261] T bcf9e97e867c4281bad84d81091b295c P b9d3e9ca17bd4d9babad18338ae019ee [term 1 FOLLOWER]: Refusing update from remote peer 045f04c1759a4d9bbf38cc06b397ba76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:43:02.904569 30198 consensus_queue.cc:1035] T bcf9e97e867c4281bad84d81091b295c P 045f04c1759a4d9bbf38cc06b397ba76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ee365d535f64fda9b04ac44dd08738c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:43:02.904690 30129 consensus_queue.cc:1035] T bcf9e97e867c4281bad84d81091b295c P 045f04c1759a4d9bbf38cc06b397ba76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d3e9ca17bd4d9babad18338ae019ee" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:43:02.922919 30195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:02.919751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:59793 (local address 127.21.143.65:36541)
0619 18:43:02.919844 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:02.919849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:02.919894 (+ 45us) server_negotiation.cc:407] Connection header received
0619 18:43:02.920009 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:02.920011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:02.920047 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:02.920151 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:43:02.920383 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:02.920992 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:02.921815 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:02.922102 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:02.922212 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:02.922475 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:43:02.922558 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:43:02.922656 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:02.922750 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 18:43:02.922802 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 18:43:02.922943 30171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:02.919698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:36541 (local address 127.21.143.67:59793)
0619 18:43:02.919846 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:02.919859 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:02.919933 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:02.920163 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:02.920167 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:02.920174 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:43:02.920273 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:02.920278 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:02.921099 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:02.921104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:02.921694 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:02.921702 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:02.922220 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:02.922224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:02.922320 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:02.922351 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:43:02.922670 (+ 319us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:43:02.922679 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 18:43:02.922749 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 18:43:02.922831 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 18:43:02.941869 29593 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:43:02.942025 29593 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:43:02.943912 29593 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3f24326df2f48d7bff275e80e8cbcfe: Sending DeleteTablet for 3 replicas of tablet bcf9e97e867c4281bad84d81091b295c
I20260619 18:43:02.944414 29731 tablet_service.cc:1514] Processing DeleteTablet for tablet bcf9e97e867c4281bad84d81091b295c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:43:02 UTC) from {username='oryx', principal='oryx/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:45454
I20260619 18:43:02.944424 30010 tablet_service.cc:1514] Processing DeleteTablet for tablet bcf9e97e867c4281bad84d81091b295c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:43:02 UTC) from {username='oryx', principal='oryx/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:43112
I20260619 18:43:02.944502 29871 tablet_service.cc:1514] Processing DeleteTablet for tablet bcf9e97e867c4281bad84d81091b295c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:43:02 UTC) from {username='oryx', principal='oryx/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:55776
I20260619 18:43:02.944585 30174 tablet_replica.cc:330] T bcf9e97e867c4281bad84d81091b295c P 045f04c1759a4d9bbf38cc06b397ba76: stopping tablet replica
I20260619 18:43:02.944697 30175 tablet_replica.cc:330] T bcf9e97e867c4281bad84d81091b295c P b9d3e9ca17bd4d9babad18338ae019ee: stopping tablet replica
I20260619 18:43:02.944720 30174 raft_consensus.cc:2229] T bcf9e97e867c4281bad84d81091b295c P 045f04c1759a4d9bbf38cc06b397ba76 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:43:02.944774 30176 tablet_replica.cc:330] T bcf9e97e867c4281bad84d81091b295c P 5ee365d535f64fda9b04ac44dd08738c: stopping tablet replica
I20260619 18:43:02.944813 30175 raft_consensus.cc:2229] T bcf9e97e867c4281bad84d81091b295c P b9d3e9ca17bd4d9babad18338ae019ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:43:02.944878 30176 raft_consensus.cc:2229] T bcf9e97e867c4281bad84d81091b295c P 5ee365d535f64fda9b04ac44dd08738c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:43:02.944892 30174 raft_consensus.cc:2258] T bcf9e97e867c4281bad84d81091b295c P 045f04c1759a4d9bbf38cc06b397ba76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:43:02.944909 30175 raft_consensus.cc:2258] T bcf9e97e867c4281bad84d81091b295c P b9d3e9ca17bd4d9babad18338ae019ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:43:02.944972 30176 raft_consensus.cc:2258] T bcf9e97e867c4281bad84d81091b295c P 5ee365d535f64fda9b04ac44dd08738c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:43:02.945122 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 29666
I20260619 18:43:02.947438 30176 ts_tablet_manager.cc:1905] T bcf9e97e867c4281bad84d81091b295c P 5ee365d535f64fda9b04ac44dd08738c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:43:02.947463 30175 ts_tablet_manager.cc:1905] T bcf9e97e867c4281bad84d81091b295c P b9d3e9ca17bd4d9babad18338ae019ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:43:02.949465 30176 ts_tablet_manager.cc:1918] T bcf9e97e867c4281bad84d81091b295c P 5ee365d535f64fda9b04ac44dd08738c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:43:02.949558 30176 log.cc:1196] T bcf9e97e867c4281bad84d81091b295c P 5ee365d535f64fda9b04ac44dd08738c: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-2/wal/wals/bcf9e97e867c4281bad84d81091b295c
I20260619 18:43:02.949791 30176 ts_tablet_manager.cc:1939] T bcf9e97e867c4281bad84d81091b295c P 5ee365d535f64fda9b04ac44dd08738c: Deleting consensus metadata
I20260619 18:43:02.950390 29578 catalog_manager.cc:4841] TS 5ee365d535f64fda9b04ac44dd08738c (127.21.143.67:38273): tablet bcf9e97e867c4281bad84d81091b295c (table test-table [id=5867ac49e80849ba9dfc49f19421b933]) successfully deleted
I20260619 18:43:02.950532 30175 ts_tablet_manager.cc:1918] T bcf9e97e867c4281bad84d81091b295c P b9d3e9ca17bd4d9babad18338ae019ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:43:02.950627 30175 log.cc:1196] T bcf9e97e867c4281bad84d81091b295c P b9d3e9ca17bd4d9babad18338ae019ee: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894542572984-22077-0/minicluster-data/ts-1/wal/wals/bcf9e97e867c4281bad84d81091b295c
I20260619 18:43:02.950889 30175 ts_tablet_manager.cc:1939] T bcf9e97e867c4281bad84d81091b295c P b9d3e9ca17bd4d9babad18338ae019ee: Deleting consensus metadata
I20260619 18:43:02.951555 29580 catalog_manager.cc:4841] TS b9d3e9ca17bd4d9babad18338ae019ee (127.21.143.66:39881): tablet bcf9e97e867c4281bad84d81091b295c (table test-table [id=5867ac49e80849ba9dfc49f19421b933]) successfully deleted
W20260619 18:43:02.953478 29580 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.143.65:36541 (error 108)
I20260619 18:43:02.953691 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 29806
I20260619 18:43:02.955960 30209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:02.954344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:36541 (local address 127.0.0.1:45488)
0619 18:43:02.955820 (+ 1476us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:02.955883 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.65:36541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1411,"thread_start_us":59,"threads_started":1}
W20260619 18:43:02.956120 29580 catalog_manager.cc:4584] TS 045f04c1759a4d9bbf38cc06b397ba76 (127.21.143.65:36541): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bcf9e97e867c4281bad84d81091b295c: Network error: Client connection negotiation failed: client connection to 127.21.143.65:36541: connect: Connection refused (error 111)
I20260619 18:43:02.959720 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 29946
I20260619 18:43:02.966842 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 29562
2026-06-19T18:43:02Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3699 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:43:03 dist-test-slave-mld5 krb5kdc[30212](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:43:03 dist-test-slave-mld5 krb5kdc[30212](info): set up 2 sockets
Jun 19 18:43:03 dist-test-slave-mld5 krb5kdc[30212](info): commencing operation
krb5kdc: starting...
W20260619 18:43:03.996114 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:43:04 dist-test-slave-mld5 krb5kdc[30212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:43:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:43:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.143.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
I20260619 18:43:04.109949 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:46717
--webserver_interface=127.21.143.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:39039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:46717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:43:04.217278 30225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:43:04.217530 30225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:43:04.217576 30225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:43:04.221735 30225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:43:04.221803 30225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:43:04.221823 30225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:43:04.221839 30225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:43:04.221855 30225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:43:04.227105 30225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:39039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:46717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:46717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:43:04.228200 30225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:43:04.229177 30225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:43:04.237607 30230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:04.237648 30225 server_base.cc:1034] running on GCE node
W20260619 18:43:04.237653 30231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:04.237764 30233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:04.238478 30225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:43:04.239614 30225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:43:04.240808 30225 hybrid_clock.cc:648] HybridClock initialized: now 1781894584240781 us; error 39 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T18:43:04Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1289 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:43:04 dist-test-slave-mld5 krb5kdc[30238](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:43:04 dist-test-slave-mld5 krb5kdc[30238](info): set up 2 sockets
Jun 19 18:43:04 dist-test-slave-mld5 krb5kdc[30238](info): commencing operation
krb5kdc: starting...
W20260619 18:43:05.296003 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:43:05 dist-test-slave-mld5 krb5kdc[30238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:43:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:43:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.143.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
I20260619 18:43:05.414124 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:40281
--webserver_interface=127.21.143.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:35789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:40281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:43:05.524245 30251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:43:05.524580 30251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:43:05.524662 30251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:43:05.528798 30251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:43:05.528898 30251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:43:05.528937 30251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:43:05.528995 30251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:43:05.529033 30251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:43:05.534559 30251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:35789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:40281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:40281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:43:05.535712 30251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:43:05.536783 30251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:43:05.544986 30256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:05.544996 30259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:05.545001 30257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:05.545236 30251 server_base.cc:1034] running on GCE node
I20260619 18:43:05.545789 30251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:43:05.547152 30251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:43:05.548329 30251 hybrid_clock.cc:648] HybridClock initialized: now 1781894585548308 us; error 51 us; skew 500 ppm
Jun 19 18:43:05 dist-test-slave-mld5 krb5kdc[30238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894585, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:43:05.551158 30251 init.cc:377] Logged in from keytab as kudu/127.21.143.126@KRBTEST.COM (short username kudu)
I20260619 18:43:05.551991 30251 webserver.cc:460] Webserver started at http://127.21.143.126:37327/ using document root <none> and password file <none>
I20260619 18:43:05.552492 30251 fs_manager.cc:359] Metadata directory not provided
I20260619 18:43:05.552548 30251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:43:05.552700 30251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:43:05.554584 30251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "7d1187986bad47bdb64745bcd1c19d2c"
format_stamp: "Formatted at 2026-06-19 18:43:05 on dist-test-slave-mld5"
server_key: "e9f6c131bfab98a7156aaea4a9fef814"
server_key_iv: "4ed7d17173f220958b5b1d5ae3b4cd60"
server_key_version: "encryptionkey@0"
I20260619 18:43:05.555068 30251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "7d1187986bad47bdb64745bcd1c19d2c"
format_stamp: "Formatted at 2026-06-19 18:43:05 on dist-test-slave-mld5"
server_key: "e9f6c131bfab98a7156aaea4a9fef814"
server_key_iv: "4ed7d17173f220958b5b1d5ae3b4cd60"
server_key_version: "encryptionkey@0"
I20260619 18:43:05.558688 30251 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260619 18:43:05.561168 30266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:05.562407 30251 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:43:05.562532 30251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "7d1187986bad47bdb64745bcd1c19d2c"
format_stamp: "Formatted at 2026-06-19 18:43:05 on dist-test-slave-mld5"
server_key: "e9f6c131bfab98a7156aaea4a9fef814"
server_key_iv: "4ed7d17173f220958b5b1d5ae3b4cd60"
server_key_version: "encryptionkey@0"
I20260619 18:43:05.562645 30251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:43:05.595595 30251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:43:05.598915 30251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:43:05.599114 30251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:43:05.606081 30251 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:40281
I20260619 18:43:05.606101 30318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:40281 every 8 connection(s)
I20260619 18:43:05.606988 30251 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:43:05.609705 30319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:43:05.610055 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 30251
I20260619 18:43:05.610199 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:43:05.610474 22077 external_mini_cluster.cc:1428] Setting key c3dceb1b9581b28d3f40848e83d4d23e
I20260619 18:43:05.616791 30319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c: Bootstrap starting.
I20260619 18:43:05.619449 30319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c: Neither blocks nor log segments found. Creating new log.
Jun 19 18:43:05 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:43:05.620302 30319 log.cc:824] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c: Log is configured to *not* fsync() on all Append() calls
I20260619 18:43:05.622417 30319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c: No bootstrap required, opened a new log
I20260619 18:43:05.624933 30322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:05.611845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42004 (local address 127.21.143.126:40281)
0619 18:43:05.612230 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:05.612248 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:05.612290 (+ 42us) server_negotiation.cc:407] Connection header received
0619 18:43:05.613068 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:05.613092 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:05.613510 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:05.613875 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:05.614512 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:05.616064 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:05.616893 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:05.617556 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:05.617709 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:05.620234 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:05.620240 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:05.620256 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:05.620285 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:05.622037 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:05.622501 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:05.622511 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:05.622538 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:05.622667 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:05.622967 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:05.622970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:05.622971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:05.623440 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:05.623648 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:05.623923 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 18:43:05.624210 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":90,"threads_started":1}
I20260619 18:43:05.626457 30319 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1187986bad47bdb64745bcd1c19d2c" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 40281 } }
I20260619 18:43:05.626701 30319 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:43:05.626781 30319 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d1187986bad47bdb64745bcd1c19d2c, State: Initialized, Role: FOLLOWER
I20260619 18:43:05.627259 30319 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1187986bad47bdb64745bcd1c19d2c" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 40281 } }
I20260619 18:43:05.627476 30319 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:43:05.627545 30319 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:43:05.627643 30319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:43:05.628545 30319 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1187986bad47bdb64745bcd1c19d2c" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 40281 } }
I20260619 18:43:05.628895 30319 leader_election.cc:302] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d1187986bad47bdb64745bcd1c19d2c; no voters:
I20260619 18:43:05.629236 30319 leader_election.cc:288] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:43:05.629351 30324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:43:05.629575 30324 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c [term 1 LEADER]: Becoming Leader. State: Replica: 7d1187986bad47bdb64745bcd1c19d2c, State: Running, Role: LEADER
I20260619 18:43:05.629941 30324 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1187986bad47bdb64745bcd1c19d2c" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 40281 } }
I20260619 18:43:05.630666 30319 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:43:05.631722 30326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d1187986bad47bdb64745bcd1c19d2c. Latest consensus state: current_term: 1 leader_uuid: "7d1187986bad47bdb64745bcd1c19d2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1187986bad47bdb64745bcd1c19d2c" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 40281 } } }
I20260619 18:43:05.631866 30326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c [sys.catalog]: This master's current role is: LEADER
I20260619 18:43:05.632156 30325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d1187986bad47bdb64745bcd1c19d2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1187986bad47bdb64745bcd1c19d2c" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 40281 } } }
I20260619 18:43:05.632248 30325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c [sys.catalog]: This master's current role is: LEADER
I20260619 18:43:05.632247 30333 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:43:05.635529 30333 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:43:05.641270 30333 catalog_manager.cc:1347] Generated new cluster ID: 7aae492752864c50894a6b2167c65ba9
I20260619 18:43:05.641346 30333 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:43:05.679255 30333 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:43:05.680603 30333 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:43:05.690232 30333 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c: Generated new TSK 0
I20260619 18:43:05.691071 30333 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.143.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab.
Entry for principal kudu/127.21.143.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab.
I20260619 18:43:05.726028 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.65:0
--local_ip_for_outbound_sockets=127.21.143.65
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:40281
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:35789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:43:05.838792 30345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:43:05.839095 30345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:43:05.839185 30345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:43:05.843325 30345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:43:05.843411 30345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:43:05.843503 30345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.65
I20260619 18:43:05.848721 30345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:35789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:40281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.65
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:43:05.849766 30345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:43:05.850831 30345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:43:05.860265 30350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:05.860384 30345 server_base.cc:1034] running on GCE node
W20260619 18:43:05.860317 30353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:05.860265 30351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:05.861021 30345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:43:05.861621 30345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:43:05.862775 30345 hybrid_clock.cc:648] HybridClock initialized: now 1781894585862760 us; error 25 us; skew 500 ppm
Jun 19 18:43:05 dist-test-slave-mld5 krb5kdc[30238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894585, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:43:05.865422 30345 init.cc:377] Logged in from keytab as kudu/127.21.143.65@KRBTEST.COM (short username kudu)
I20260619 18:43:05.866323 30345 webserver.cc:460] Webserver started at http://127.21.143.65:46209/ using document root <none> and password file <none>
I20260619 18:43:05.866974 30345 fs_manager.cc:359] Metadata directory not provided
I20260619 18:43:05.867064 30345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:43:05.867280 30345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:43:05.869313 30345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-0/data/instance:
uuid: "9860b669cec2428295efe223c5886357"
format_stamp: "Formatted at 2026-06-19 18:43:05 on dist-test-slave-mld5"
server_key: "6fd277bcb2c86d1da494a6ddd992d2c5"
server_key_iv: "cd9572ccac07b920c6d798665bcad695"
server_key_version: "encryptionkey@0"
I20260619 18:43:05.869879 30345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance:
uuid: "9860b669cec2428295efe223c5886357"
format_stamp: "Formatted at 2026-06-19 18:43:05 on dist-test-slave-mld5"
server_key: "6fd277bcb2c86d1da494a6ddd992d2c5"
server_key_iv: "cd9572ccac07b920c6d798665bcad695"
server_key_version: "encryptionkey@0"
I20260619 18:43:05.873488 30345 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:43:05.875830 30360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:05.876956 30345 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:43:05.877087 30345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-0/wal
uuid: "9860b669cec2428295efe223c5886357"
format_stamp: "Formatted at 2026-06-19 18:43:05 on dist-test-slave-mld5"
server_key: "6fd277bcb2c86d1da494a6ddd992d2c5"
server_key_iv: "cd9572ccac07b920c6d798665bcad695"
server_key_version: "encryptionkey@0"
I20260619 18:43:05.877197 30345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:43:05.910854 30345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:43:05.913789 30345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:43:05.914004 30345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:43:05.914623 30345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:43:05.915470 30345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:43:05.915541 30345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:05.915604 30345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:43:05.915645 30345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:05.924301 30345 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.65:36155
I20260619 18:43:05.924341 30473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.65:36155 every 8 connection(s)
I20260619 18:43:05.925220 30345 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
I20260619 18:43:05.932866 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 30345
I20260619 18:43:05.933022 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance
I20260619 18:43:05.933367 22077 external_mini_cluster.cc:1428] Setting key 45f85d9698e247378ebe8cf7f3b8f8ef
Jun 19 18:43:05 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894585, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:43:05.938843 30322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:05.926936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:59355 (local address 127.21.143.126:40281)
0619 18:43:05.927113 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:05.927118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:05.927991 (+ 873us) server_negotiation.cc:407] Connection header received
0619 18:43:05.929205 (+ 1214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:05.929207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:05.929253 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:05.929324 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:05.930459 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:05.930977 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:05.932247 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:05.932498 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:05.932576 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:05.935118 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:05.935123 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:05.935125 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:05.935154 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:05.936749 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:05.937356 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:05.937358 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:05.937359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:05.937428 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:05.937808 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:05.937810 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:05.937811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:05.937970 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:05.938042 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:05.938629 (+ 587us) server_negotiation.cc:299] Negotiation successful
0619 18:43:05.938724 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 18:43:05.940066 30476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:05.927407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.65:59355)
0619 18:43:05.927841 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:05.927883 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:05.929019 (+ 1136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:05.929488 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:05.929494 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:05.929820 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:05.930338 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:05.930353 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:05.931117 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:05.931122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:05.932101 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:05.932110 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:05.932618 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:05.932621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:05.932750 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:05.933312 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:05.933334 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:05.934958 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:43:05.936862 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:05.936871 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:05.936887 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:05.937253 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:05.937522 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:05.937526 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:05.937529 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:05.937715 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:05.938072 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:43:05.938081 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:43:05.938391 (+ 310us) client_negotiation.cc:769] Sending connection context
0619 18:43:05.938592 (+ 201us) client_negotiation.cc:240] Negotiation successful
0619 18:43:05.938937 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":112,"threads_started":1}
I20260619 18:43:05.941332 30474 heartbeater.cc:347] Connected to a master server at 127.21.143.126:40281
I20260619 18:43:05.941563 30474 heartbeater.cc:464] Registering TS with master...
I20260619 18:43:05.942101 30474 heartbeater.cc:511] Master 127.21.143.126:40281 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.143.66@KRBTEST.COM; defaulting to no policy
I20260619 18:43:05.943645 30283 ts_manager.cc:194] Registered new tserver with Master: 9860b669cec2428295efe223c5886357 (127.21.143.65:36155)
I20260619 18:43:05.944844 30283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.65@KRBTEST.COM'} at 127.21.143.65:59355
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab.
Entry for principal kudu/127.21.143.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab.
I20260619 18:43:05.960422 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.66:0
--local_ip_for_outbound_sockets=127.21.143.66
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:40281
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:35789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:43:06.068179 30479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:43:06.068434 30479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:43:06.068476 30479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:43:06.072361 30479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:43:06.072413 30479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:43:06.072469 30479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.66
I20260619 18:43:06.077446 30479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:35789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:40281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.66
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:43:06.078382 30479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:43:06.079392 30479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:43:06.088634 30484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:06.088691 30479 server_base.cc:1034] running on GCE node
W20260619 18:43:06.088641 30485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:06.088649 30487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:06.089309 30479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:43:06.089915 30479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:43:06.091100 30479 hybrid_clock.cc:648] HybridClock initialized: now 1781894586091087 us; error 34 us; skew 500 ppm
Jun 19 18:43:06 dist-test-slave-mld5 krb5kdc[30238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894586, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:43:06.093941 30479 init.cc:377] Logged in from keytab as kudu/127.21.143.66@KRBTEST.COM (short username kudu)
I20260619 18:43:06.094825 30479 webserver.cc:460] Webserver started at http://127.21.143.66:42513/ using document root <none> and password file <none>
I20260619 18:43:06.095453 30479 fs_manager.cc:359] Metadata directory not provided
I20260619 18:43:06.095542 30479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:43:06.095767 30479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:43:06.097781 30479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-1/data/instance:
uuid: "434941e6a4964e399239a4e4f1a2e5e6"
format_stamp: "Formatted at 2026-06-19 18:43:06 on dist-test-slave-mld5"
server_key: "ca807572f58c839ff5c90272a93f9b75"
server_key_iv: "9b203cd864668b308b6bc1fc9db750fe"
server_key_version: "encryptionkey@0"
I20260619 18:43:06.098347 30479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance:
uuid: "434941e6a4964e399239a4e4f1a2e5e6"
format_stamp: "Formatted at 2026-06-19 18:43:06 on dist-test-slave-mld5"
server_key: "ca807572f58c839ff5c90272a93f9b75"
server_key_iv: "9b203cd864668b308b6bc1fc9db750fe"
server_key_version: "encryptionkey@0"
I20260619 18:43:06.102020 30479 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 18:43:06.104388 30494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:06.105638 30479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:43:06.105767 30479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-1/wal
uuid: "434941e6a4964e399239a4e4f1a2e5e6"
format_stamp: "Formatted at 2026-06-19 18:43:06 on dist-test-slave-mld5"
server_key: "ca807572f58c839ff5c90272a93f9b75"
server_key_iv: "9b203cd864668b308b6bc1fc9db750fe"
server_key_version: "encryptionkey@0"
I20260619 18:43:06.105875 30479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:43:06.130971 30479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:43:06.134451 30479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:43:06.134660 30479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:43:06.135248 30479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:43:06.136125 30479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:43:06.136194 30479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:06.136265 30479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:43:06.136317 30479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:06.145399 30479 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.66:43993
I20260619 18:43:06.145417 30607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.66:43993 every 8 connection(s)
I20260619 18:43:06.146327 30479 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
I20260619 18:43:06.146956 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 30479
I20260619 18:43:06.147085 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance
I20260619 18:43:06.147411 22077 external_mini_cluster.cc:1428] Setting key e0aa5f58dfa6a9b5dfe328588315b15f
WARNING: no policy specified for kudu/127.21.143.67@KRBTEST.COM; defaulting to no policy
Jun 19 18:43:06 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894586, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.67@KRBTEST.COM" created.
I20260619 18:43:06.162243 30322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:06.148381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:49317 (local address 127.21.143.126:40281)
0619 18:43:06.148523 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:06.148528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:06.149687 (+ 1159us) server_negotiation.cc:407] Connection header received
0619 18:43:06.151387 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:06.151390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:06.151444 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:06.151538 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:06.153116 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:06.153716 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:06.155585 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:06.155797 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:06.155874 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:06.158647 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:06.158650 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:06.158652 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:06.158673 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:06.160282 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:06.160756 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:06.160761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:06.160763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:06.160844 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:06.161175 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:06.161178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:06.161180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:06.161365 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:06.161477 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:06.161958 (+ 481us) server_negotiation.cc:299] Negotiation successful
0619 18:43:06.162074 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:43:06.163087 30610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:06.148966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.66:49317)
0619 18:43:06.149517 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:06.149576 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:06.151157 (+ 1581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:06.151717 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:06.151724 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:06.152251 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:06.152935 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:06.152958 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:06.153851 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:06.153857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:06.155472 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:06.155484 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:06.155960 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:06.155973 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:06.156198 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:06.156801 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:06.156821 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:06.158499 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:43:06.160390 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:06.160396 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:06.160409 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:06.160653 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:06.160933 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:06.160938 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:06.160939 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:06.161073 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:06.161486 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:43:06.161494 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:43:06.161726 (+ 232us) client_negotiation.cc:769] Sending connection context
0619 18:43:06.161919 (+ 193us) client_negotiation.cc:240] Negotiation successful
0619 18:43:06.162220 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":125,"threads_started":1}
I20260619 18:43:06.164283 30608 heartbeater.cc:347] Connected to a master server at 127.21.143.126:40281
I20260619 18:43:06.164578 30608 heartbeater.cc:464] Registering TS with master...
I20260619 18:43:06.165242 30608 heartbeater.cc:511] Master 127.21.143.126:40281 requested a full tablet report, sending...
I20260619 18:43:06.166477 30283 ts_manager.cc:194] Registered new tserver with Master: 434941e6a4964e399239a4e4f1a2e5e6 (127.21.143.66:43993)
I20260619 18:43:06.167081 30283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.66@KRBTEST.COM'} at 127.21.143.66:49317
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab.
Entry for principal kudu/127.21.143.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab.
I20260619 18:43:06.176100 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.67:0
--local_ip_for_outbound_sockets=127.21.143.67
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:40281
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:35789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:43:06.288995 30613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:43:06.289220 30613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:43:06.289294 30613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:43:06.293238 30613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:43:06.293286 30613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:43:06.293372 30613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.67
I20260619 18:43:06.298426 30613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:35789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:40281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.67
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:43:06.299481 30613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:43:06.300439 30613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:43:06.309422 30619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:06.309433 30618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:06.309419 30621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:06.309728 30613 server_base.cc:1034] running on GCE node
I20260619 18:43:06.310175 30613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:43:06.310855 30613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:43:06.312078 30613 hybrid_clock.cc:648] HybridClock initialized: now 1781894586312057 us; error 38 us; skew 500 ppm
Jun 19 18:43:06 dist-test-slave-mld5 krb5kdc[30238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894586, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:43:06.314657 30613 init.cc:377] Logged in from keytab as kudu/127.21.143.67@KRBTEST.COM (short username kudu)
I20260619 18:43:06.315557 30613 webserver.cc:460] Webserver started at http://127.21.143.67:44041/ using document root <none> and password file <none>
I20260619 18:43:06.316098 30613 fs_manager.cc:359] Metadata directory not provided
I20260619 18:43:06.316185 30613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:43:06.316406 30613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:43:06.318312 30613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-2/data/instance:
uuid: "a401f763bca54418aed2390b4ceb4706"
format_stamp: "Formatted at 2026-06-19 18:43:06 on dist-test-slave-mld5"
server_key: "a7c79d749364703fd5d8a2416a197c4b"
server_key_iv: "124363cade988e2e5c5d845bb0651be3"
server_key_version: "encryptionkey@0"
I20260619 18:43:06.318856 30613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance:
uuid: "a401f763bca54418aed2390b4ceb4706"
format_stamp: "Formatted at 2026-06-19 18:43:06 on dist-test-slave-mld5"
server_key: "a7c79d749364703fd5d8a2416a197c4b"
server_key_iv: "124363cade988e2e5c5d845bb0651be3"
server_key_version: "encryptionkey@0"
I20260619 18:43:06.322472 30613 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 18:43:06.324822 30628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:06.325977 30613 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:43:06.326112 30613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-2/wal
uuid: "a401f763bca54418aed2390b4ceb4706"
format_stamp: "Formatted at 2026-06-19 18:43:06 on dist-test-slave-mld5"
server_key: "a7c79d749364703fd5d8a2416a197c4b"
server_key_iv: "124363cade988e2e5c5d845bb0651be3"
server_key_version: "encryptionkey@0"
I20260619 18:43:06.326217 30613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:43:06.355895 30613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:43:06.358704 30613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:43:06.358912 30613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:43:06.359580 30613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:43:06.360423 30613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:43:06.360495 30613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:06.360565 30613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:43:06.360613 30613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:06.370126 30613 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.67:39005
I20260619 18:43:06.370150 30741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.67:39005 every 8 connection(s)
I20260619 18:43:06.371073 30613 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
I20260619 18:43:06.374954 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 30613
I20260619 18:43:06.375133 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance
I20260619 18:43:06.375545 22077 external_mini_cluster.cc:1428] Setting key 8dedb75eb94e5a15fff2886b40335661
Jun 19 18:43:06 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894586, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:43:06.385579 30322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:06.373019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:58487 (local address 127.21.143.126:40281)
0619 18:43:06.373167 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:06.373170 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:06.374243 (+ 1073us) server_negotiation.cc:407] Connection header received
0619 18:43:06.375768 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:06.375771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:06.375830 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:06.375916 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:06.377511 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:06.378138 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:06.379339 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:06.379680 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:06.379763 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:06.382410 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:06.382413 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:06.382415 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:06.382440 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:06.383756 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:06.384226 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:06.384228 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:06.384229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:06.384354 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:06.384652 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:06.384655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:06.384656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:06.384835 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:06.384907 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:06.385353 (+ 446us) server_negotiation.cc:299] Negotiation successful
0619 18:43:06.385448 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 18:43:06.386404 30744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:06.373609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.67:58487)
0619 18:43:06.374083 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:06.374129 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:06.375536 (+ 1407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:06.376166 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:06.376176 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:06.376717 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:06.377355 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:06.377369 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:06.378253 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:06.378256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:06.379196 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:06.379205 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:06.379869 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:06.379873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:06.380015 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:06.380686 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:06.380702 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:06.382242 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:43:06.383857 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:06.383862 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:06.383874 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:06.384140 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:06.384434 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:06.384436 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:06.384437 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:06.384573 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:06.384925 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:43:06.384930 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:43:06.385157 (+ 227us) client_negotiation.cc:769] Sending connection context
0619 18:43:06.385311 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 18:43:06.385564 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":130,"threads_started":1}
I20260619 18:43:06.387588 30742 heartbeater.cc:347] Connected to a master server at 127.21.143.126:40281
I20260619 18:43:06.387853 30742 heartbeater.cc:464] Registering TS with master...
I20260619 18:43:06.388433 30742 heartbeater.cc:511] Master 127.21.143.126:40281 requested a full tablet report, sending...
I20260619 18:43:06.389607 30283 ts_manager.cc:194] Registered new tserver with Master: a401f763bca54418aed2390b4ceb4706 (127.21.143.67:39005)
I20260619 18:43:06.390162 30283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.67@KRBTEST.COM'} at 127.21.143.67:58487
I20260619 18:43:06.401031 22077 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.21.143.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.143.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.143.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.65.keytab.
Entry for principal oryx/127.21.143.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.65.keytab.
WARNING: no policy specified for oryx/127.21.143.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.143.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.143.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.66.keytab.
Entry for principal oryx/127.21.143.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.66.keytab.
WARNING: no policy specified for oryx/127.21.143.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.143.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.143.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.67.keytab.
Entry for principal oryx/127.21.143.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.67.keytab.
I20260619 18:43:06.465059 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 30345
I20260619 18:43:06.471706 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 30479
I20260619 18:43:06.477075 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 30613
I20260619 18:43:06.482059 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 30251
I20260619 18:43:06.488245 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:40281
--webserver_interface=127.21.143.126
--webserver_port=37327
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:35789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:40281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:43:06.599131 30751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:43:06.599431 30751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:43:06.599472 30751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:43:06.603463 30751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:43:06.603509 30751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:43:06.603534 30751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:43:06.603582 30751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:43:06.603616 30751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:43:06.608837 30751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:35789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:40281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:40281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=37327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:43:06.609822 30751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:43:06.610819 30751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:43:06.619031 30757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:06.619155 30751 server_base.cc:1034] running on GCE node
W20260619 18:43:06.619043 30759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:06.619026 30756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:06.619961 30751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:43:06.621084 30751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:43:06.622272 30751 hybrid_clock.cc:648] HybridClock initialized: now 1781894586622259 us; error 33 us; skew 500 ppm
Jun 19 18:43:06 dist-test-slave-mld5 krb5kdc[30238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894586, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:43:06.624953 30751 init.cc:377] Logged in from keytab as kudu/127.21.143.126@KRBTEST.COM (short username kudu)
I20260619 18:43:06.625845 30751 webserver.cc:460] Webserver started at http://127.21.143.126:37327/ using document root <none> and password file <none>
I20260619 18:43:06.626446 30751 fs_manager.cc:359] Metadata directory not provided
I20260619 18:43:06.626551 30751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:43:06.630247 30751 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:43:06.632288 30766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:06.633402 30751 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:43:06.633544 30751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "7d1187986bad47bdb64745bcd1c19d2c"
format_stamp: "Formatted at 2026-06-19 18:43:05 on dist-test-slave-mld5"
server_key: "e9f6c131bfab98a7156aaea4a9fef814"
server_key_iv: "4ed7d17173f220958b5b1d5ae3b4cd60"
server_key_version: "encryptionkey@0"
I20260619 18:43:06.634059 30751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:43:06.651146 30751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:43:06.653683 30751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:43:06.653882 30751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:43:06.660972 30751 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:40281
I20260619 18:43:06.660995 30818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:40281 every 8 connection(s)
I20260619 18:43:06.662000 30751 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:43:06.666013 30819 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:43:06.667574 30819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c: Bootstrap starting.
I20260619 18:43:06.670324 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 30751
I20260619 18:43:06.670781 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.65:36155
--local_ip_for_outbound_sockets=127.21.143.65
--tserver_master_addrs=127.21.143.126:40281
--webserver_port=46209
--webserver_interface=127.21.143.65
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:35789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.143.65
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.65.keytab with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:43:06.683159 30819 log.cc:824] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c: Log is configured to *not* fsync() on all Append() calls
I20260619 18:43:06.689455 30819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:43:06.689803 30819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c: Bootstrap complete.
I20260619 18:43:06.694015 30819 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1187986bad47bdb64745bcd1c19d2c" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 40281 } }
I20260619 18:43:06.694468 30819 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d1187986bad47bdb64745bcd1c19d2c, State: Initialized, Role: FOLLOWER
I20260619 18:43:06.694983 30819 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1187986bad47bdb64745bcd1c19d2c" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 40281 } }
I20260619 18:43:06.695122 30819 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:43:06.695210 30819 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:43:06.695363 30819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c [term 1 FOLLOWER]: Advancing to term 2
I20260619 18:43:06.697357 30819 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1187986bad47bdb64745bcd1c19d2c" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 40281 } }
I20260619 18:43:06.697662 30819 leader_election.cc:302] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d1187986bad47bdb64745bcd1c19d2c; no voters:
I20260619 18:43:06.697991 30819 leader_election.cc:288] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 18:43:06.698107 30823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c [term 2 FOLLOWER]: Leader election won for term 2
I20260619 18:43:06.698292 30823 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c [term 2 LEADER]: Becoming Leader. State: Replica: 7d1187986bad47bdb64745bcd1c19d2c, State: Running, Role: LEADER
I20260619 18:43:06.698658 30823 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1187986bad47bdb64745bcd1c19d2c" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 40281 } }
I20260619 18:43:06.699180 30819 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:43:06.700091 30825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d1187986bad47bdb64745bcd1c19d2c. Latest consensus state: current_term: 2 leader_uuid: "7d1187986bad47bdb64745bcd1c19d2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1187986bad47bdb64745bcd1c19d2c" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 40281 } } }
I20260619 18:43:06.700137 30824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7d1187986bad47bdb64745bcd1c19d2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1187986bad47bdb64745bcd1c19d2c" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 40281 } } }
I20260619 18:43:06.700316 30824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c [sys.catalog]: This master's current role is: LEADER
I20260619 18:43:06.700318 30825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c [sys.catalog]: This master's current role is: LEADER
I20260619 18:43:06.700893 30832 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:43:06.704080 30832 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:43:06.704843 30832 catalog_manager.cc:1259] Loaded cluster ID: 7aae492752864c50894a6b2167c65ba9
I20260619 18:43:06.704905 30832 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:43:06.707271 30832 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:43:06.707923 30832 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7d1187986bad47bdb64745bcd1c19d2c: Loaded TSK: 0
I20260619 18:43:06.709098 30832 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 18:43:06.792852 30821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:43:06.793210 30821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:43:06.793308 30821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:43:06.797693 30821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:43:06.797763 30821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:43:06.797887 30821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.65
I20260619 18:43:06.803934 30821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:35789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.65.keytab
--principal=oryx/127.21.143.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.65:36155
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.143.65
--webserver_port=46209
--tserver_master_addrs=127.21.143.126:40281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.65
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:43:06.804963 30821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:43:06.806028 30821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:43:06.814864 30846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:06.814878 30845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:06.815102 30848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:06.816720 30821 server_base.cc:1034] running on GCE node
I20260619 18:43:06.817111 30821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:43:06.817740 30821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:43:06.818907 30821 hybrid_clock.cc:648] HybridClock initialized: now 1781894586818882 us; error 41 us; skew 500 ppm
Jun 19 18:43:06 dist-test-slave-mld5 krb5kdc[30238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894586, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:43:06.821774 30821 init.cc:377] Logged in from keytab as oryx/127.21.143.65@KRBTEST.COM (short username oryx)
I20260619 18:43:06.822683 30821 webserver.cc:460] Webserver started at http://127.21.143.65:46209/ using document root <none> and password file <none>
I20260619 18:43:06.823236 30821 fs_manager.cc:359] Metadata directory not provided
I20260619 18:43:06.823364 30821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:43:06.827283 30821 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260619 18:43:06.829501 30855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:06.830742 30821 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260619 18:43:06.830879 30821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-0/wal
uuid: "9860b669cec2428295efe223c5886357"
format_stamp: "Formatted at 2026-06-19 18:43:05 on dist-test-slave-mld5"
server_key: "6fd277bcb2c86d1da494a6ddd992d2c5"
server_key_iv: "cd9572ccac07b920c6d798665bcad695"
server_key_version: "encryptionkey@0"
I20260619 18:43:06.831473 30821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:43:06.844451 30821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:43:06.847612 30821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:43:06.847813 30821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:43:06.848394 30821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:43:06.849236 30821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:43:06.849309 30821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:06.849376 30821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:43:06.849428 30821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:06.858040 30821 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.65:36155
I20260619 18:43:06.858064 30968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.65:36155 every 8 connection(s)
I20260619 18:43:06.859005 30821 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
I20260619 18:43:06.866958 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 30821
I20260619 18:43:06.867434 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.66:43993
--local_ip_for_outbound_sockets=127.21.143.66
--tserver_master_addrs=127.21.143.126:40281
--webserver_port=42513
--webserver_interface=127.21.143.66
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:35789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.143.66
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.66.keytab with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:43:06 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:06 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:06.871886 30972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:06.861157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.65:54303)
0619 18:43:06.861610 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:06.861660 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:06.862846 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:06.863949 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:06.863956 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:06.864280 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:06.864709 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:06.864722 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:06.865872 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:06.865874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:06.866792 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:06.866805 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:06.867611 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:06.867618 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:06.867841 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:06.868444 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:06.868469 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:06.871090 (+ 2621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":138,"threads_started":1}
W20260619 18:43:06.872316 30971 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:06.860988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:54303 (local address 127.21.143.126:40281)
0619 18:43:06.861402 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:06.861418 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:06.861782 (+ 364us) server_negotiation.cc:407] Connection header received
0619 18:43:06.863293 (+ 1511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:06.863328 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:06.863599 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:06.863857 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:06.864846 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:06.865766 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:06.866923 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:06.867438 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:06.867630 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:06.871372 (+ 3742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.65:54303: BlockingRecv error: recv got EOF from 127.21.143.65:54303 (error 108)
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":114,"threads_started":1}
W20260619 18:43:06.872362 30969 heartbeater.cc:644] Failed to heartbeat to 127.21.143.126:40281 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.143.126:40281: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Jun 19 18:43:06 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:06 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:06.878870 30972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:06.872751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.65:34211)
0619 18:43:06.872946 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:06.872957 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:06.873017 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:06.873315 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:06.873318 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:06.873448 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:06.873533 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:06.873538 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:06.874650 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:06.874652 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:06.875512 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:06.875522 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:06.876228 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:06.876232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:06.876326 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:06.876827 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:06.876855 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:06.878738 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136}
W20260619 18:43:06.878979 30971 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:06.872844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:34211 (local address 127.21.143.126:40281)
0619 18:43:06.872985 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:06.872991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:06.873008 (+ 17us) server_negotiation.cc:407] Connection header received
0619 18:43:06.873107 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:06.873113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:06.873169 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:06.873278 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:06.873650 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:06.874517 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:06.875626 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:06.876102 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:06.876213 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:06.878851 (+ 2638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.65:34211: BlockingRecv error: recv got EOF from 127.21.143.65:34211 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 18:43:06 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:06 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:06.884298 30972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:06.879607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.65:58251)
0619 18:43:06.879693 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:06.879707 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:06.879775 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:06.880008 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:06.880011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:06.880156 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:06.880220 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:06.880224 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:06.880910 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:06.880915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:06.881469 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:06.881480 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:06.881929 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:06.881932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:06.882008 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:06.882463 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:06.882473 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:06.884158 (+ 1685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260619 18:43:06.884382 30971 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:06.879699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:58251 (local address 127.21.143.126:40281)
0619 18:43:06.879802 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:06.879809 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:06.879834 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:43:06.879875 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:06.879877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:06.879910 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:06.879994 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:06.880328 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:06.880805 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:06.881606 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:06.881822 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:06.881928 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:06.884274 (+ 2346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.65:58251: BlockingRecv error: recv got EOF from 127.21.143.65:58251 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260619 18:43:06.884720 30969 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 18:43:06.978340 30973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:43:06.978848 30973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:43:06.978916 30973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:43:06.983186 30973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:43:06.983263 30973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:43:06.983342 30973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.66
I20260619 18:43:06.989018 30973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:35789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.66.keytab
--principal=oryx/127.21.143.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.66:43993
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.143.66
--webserver_port=42513
--tserver_master_addrs=127.21.143.126:40281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.66
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:43:06.990216 30973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:43:06.991493 30973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:43:07.001814 30978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:07.001811 30981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:07.001811 30979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:07.002012 30973 server_base.cc:1034] running on GCE node
I20260619 18:43:07.002460 30973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:43:07.003126 30973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:43:07.004333 30973 hybrid_clock.cc:648] HybridClock initialized: now 1781894587004332 us; error 61 us; skew 500 ppm
Jun 19 18:43:07 dist-test-slave-mld5 krb5kdc[30238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894587, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:43:07.007085 30973 init.cc:377] Logged in from keytab as oryx/127.21.143.66@KRBTEST.COM (short username oryx)
I20260619 18:43:07.008064 30973 webserver.cc:460] Webserver started at http://127.21.143.66:42513/ using document root <none> and password file <none>
I20260619 18:43:07.008673 30973 fs_manager.cc:359] Metadata directory not provided
I20260619 18:43:07.008778 30973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:43:07.012689 30973 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260619 18:43:07.014642 30988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:07.015877 30973 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:43:07.016011 30973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-1/wal
uuid: "434941e6a4964e399239a4e4f1a2e5e6"
format_stamp: "Formatted at 2026-06-19 18:43:06 on dist-test-slave-mld5"
server_key: "ca807572f58c839ff5c90272a93f9b75"
server_key_iv: "9b203cd864668b308b6bc1fc9db750fe"
server_key_version: "encryptionkey@0"
I20260619 18:43:07.016573 30973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:43:07.029524 30973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:43:07.032433 30973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:43:07.032637 30973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:43:07.033226 30973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:43:07.034049 30973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:43:07.034130 30973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:07.034197 30973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:43:07.034242 30973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:07.043478 30973 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.66:43993
I20260619 18:43:07.043571 31101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.66:43993 every 8 connection(s)
I20260619 18:43:07.044618 30973 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
Jun 19 18:43:07 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:07.054651 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 30973
Jun 19 18:43:07 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:07.055105 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.67:39005
--local_ip_for_outbound_sockets=127.21.143.67
--tserver_master_addrs=127.21.143.126:40281
--webserver_port=44041
--webserver_interface=127.21.143.67
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:35789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.143.67
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.67.keytab with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:43:07.056030 30971 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:07.046457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:34469 (local address 127.21.143.126:40281)
0619 18:43:07.046608 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:07.046611 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:07.047515 (+ 904us) server_negotiation.cc:407] Connection header received
0619 18:43:07.048859 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:07.048863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:07.048924 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:07.049034 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:07.050212 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:07.050744 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:07.051986 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:07.052229 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:07.052337 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:07.055853 (+ 3516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.66:34469: BlockingRecv error: recv got EOF from 127.21.143.66:34469 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 18:43:07.056489 31104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:07.046861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.66:34469)
0619 18:43:07.047353 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:07.047413 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:07.048635 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:07.049230 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:07.049237 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:07.049587 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:07.050056 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:07.050078 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:07.050871 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:07.050874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:07.051850 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:07.051859 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:07.052378 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:07.052382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:07.052551 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:07.053090 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:07.053106 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:07.055667 (+ 2561us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":114,"threads_started":1}
W20260619 18:43:07.056974 31102 heartbeater.cc:644] Failed to heartbeat to 127.21.143.126:40281 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.143.126:40281: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Jun 19 18:43:07 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:07 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:07.062364 31104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:07.057413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.66:37985)
0619 18:43:07.057503 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:07.057518 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:07.057616 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:07.057874 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:07.057877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:07.058033 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:07.058114 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:07.058119 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:07.058824 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:07.058830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:07.059434 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:07.059442 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:07.059866 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:07.059869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:07.059959 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:07.060442 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:07.060454 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:07.062181 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 18:43:07.062371 30971 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:07.057537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:37985 (local address 127.21.143.126:40281)
0619 18:43:07.057682 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:07.057687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:07.057699 (+ 12us) server_negotiation.cc:407] Connection header received
0619 18:43:07.057742 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:07.057744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:07.057778 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:07.057860 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:07.058225 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:07.058723 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:07.059545 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:07.059762 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:07.059866 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:07.062285 (+ 2419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.66:37985: BlockingRecv error: recv got EOF from 127.21.143.66:37985 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 18:43:07 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:07 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:07.068349 31104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:07.063026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.66:54195)
0619 18:43:07.063214 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:07.063229 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:07.063353 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:07.063589 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:07.063595 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:07.063767 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:07.063875 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:07.063884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:07.064575 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:07.064580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:07.065586 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:07.065599 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:07.065998 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:07.066002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:07.066086 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:07.066507 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:07.066522 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:07.068224 (+ 1702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109}
W20260619 18:43:07.068428 30971 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:07.063098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:54195 (local address 127.21.143.126:40281)
0619 18:43:07.063197 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:07.063200 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:07.063287 (+ 87us) server_negotiation.cc:407] Connection header received
0619 18:43:07.063440 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:07.063442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:07.063476 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:07.063540 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:07.063995 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:07.064471 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:07.065705 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:07.065896 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:07.065972 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:07.068322 (+ 2350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.66:54195: BlockingRecv error: recv got EOF from 127.21.143.66:54195 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260619 18:43:07.068608 31102 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 18:43:07.167817 31105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:43:07.168135 31105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:43:07.168213 31105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:43:07.172534 31105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:43:07.172731 31105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:43:07.172897 31105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.67
I20260619 18:43:07.178565 31105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:35789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/krb5kdc/oryx_127.21.143.67.keytab
--principal=oryx/127.21.143.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.67:39005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.143.67
--webserver_port=44041
--tserver_master_addrs=127.21.143.126:40281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.67
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:43:07.179842 31105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:43:07.181010 31105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:43:07.191138 31113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:07.191133 31111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:07.191125 31105 server_base.cc:1034] running on GCE node
W20260619 18:43:07.191100 31110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:07.191815 31105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:43:07.192510 31105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:43:07.193692 31105 hybrid_clock.cc:648] HybridClock initialized: now 1781894587193643 us; error 62 us; skew 500 ppm
Jun 19 18:43:07 dist-test-slave-mld5 krb5kdc[30238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894587, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.143.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:43:07.196453 31105 init.cc:377] Logged in from keytab as oryx/127.21.143.67@KRBTEST.COM (short username oryx)
I20260619 18:43:07.197357 31105 webserver.cc:460] Webserver started at http://127.21.143.67:44041/ using document root <none> and password file <none>
I20260619 18:43:07.197944 31105 fs_manager.cc:359] Metadata directory not provided
I20260619 18:43:07.198055 31105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:43:07.202260 31105 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:43:07.204272 31120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:07.205484 31105 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:43:07.205621 31105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-2/wal
uuid: "a401f763bca54418aed2390b4ceb4706"
format_stamp: "Formatted at 2026-06-19 18:43:06 on dist-test-slave-mld5"
server_key: "a7c79d749364703fd5d8a2416a197c4b"
server_key_iv: "124363cade988e2e5c5d845bb0651be3"
server_key_version: "encryptionkey@0"
I20260619 18:43:07.206136 31105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:43:07.218308 31105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:43:07.221432 31105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:43:07.221614 31105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:43:07.222162 31105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:43:07.223026 31105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:43:07.223093 31105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:07.223156 31105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:43:07.223188 31105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:07.232575 31105 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.67:39005
I20260619 18:43:07.232595 31233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.67:39005 every 8 connection(s)
I20260619 18:43:07.233599 31105 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
I20260619 18:43:07.241917 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 31105
Jun 19 18:43:07 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:07 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 18:43:07.245391 30971 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:07.235540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:59393 (local address 127.21.143.126:40281)
0619 18:43:07.235690 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:07.235695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:07.236732 (+ 1037us) server_negotiation.cc:407] Connection header received
0619 18:43:07.238220 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:07.238224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:07.238281 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:07.238364 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:07.239627 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:07.240158 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:07.241324 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:07.241629 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:07.241744 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:07.245262 (+ 3518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.67:59393: BlockingRecv error: recv got EOF from 127.21.143.67:59393 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 18:43:07.246021 31236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:07.236090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.67:59393)
0619 18:43:07.236543 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:07.236586 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:07.238010 (+ 1424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:07.238565 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:07.238572 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:07.238927 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:07.239484 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:07.239500 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:07.240285 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:07.240289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:07.241197 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:07.241206 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:07.241781 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:07.241785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:07.241934 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:07.242601 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:07.242618 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:07.245120 (+ 2502us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":113,"threads_started":1}
W20260619 18:43:07.246479 31234 heartbeater.cc:644] Failed to heartbeat to 127.21.143.126:40281 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.143.126:40281: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Jun 19 18:43:07 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:07 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:07.252117 31236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:07.246899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.67:55827)
0619 18:43:07.246997 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:07.247011 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:07.247079 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:07.247286 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:07.247289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:07.247464 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:07.247597 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:07.247606 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:07.248385 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:07.248387 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:07.248922 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:07.248930 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:07.249529 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:07.249532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:07.249599 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:07.250004 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:07.250021 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:07.251983 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 18:43:07.252218 30971 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:07.246971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:55827 (local address 127.21.143.126:40281)
0619 18:43:07.247067 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:07.247071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:07.247086 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:43:07.247141 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:07.247143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:07.247175 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:07.247237 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:07.247713 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:07.248276 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:07.249036 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:07.249397 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:07.249512 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:07.252113 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.67:55827: BlockingRecv error: recv got EOF from 127.21.143.67:55827 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 18:43:07.252457 31238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:07.242989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52424 (local address 127.21.143.126:40281)
0619 18:43:07.243226 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:07.243229 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:07.243239 (+ 10us) server_negotiation.cc:407] Connection header received
0619 18:43:07.243269 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:07.243271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:07.243361 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:07.243475 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:07.244266 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:07.244845 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:07.245948 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:07.246249 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:07.246344 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:07.248122 (+ 1778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:07.248129 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:07.248147 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:07.248176 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:07.250093 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:07.250573 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:07.250580 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:07.250586 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:07.250741 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:07.251065 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:07.251069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:07.251073 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:07.251572 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:07.251766 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:07.252051 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 18:43:07.252285 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":93,"threads_started":1}
Jun 19 18:43:07 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:07 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:07.257889 31236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:07.252887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.67:40049)
0619 18:43:07.252969 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:07.252980 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:07.253039 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:07.253700 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:07.253703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:07.253854 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:07.253924 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:07.253929 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:07.254739 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:07.254745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:07.255381 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:07.255389 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:07.255828 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:07.255832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:07.255919 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:07.256255 (+ 336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:07.256266 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:07.257765 (+ 1499us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 18:43:07.257979 31238 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:07.253388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:40049 (local address 127.21.143.126:40281)
0619 18:43:07.253520 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:07.253523 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:07.253533 (+ 10us) server_negotiation.cc:407] Connection header received
0619 18:43:07.253561 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:07.253563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:07.253594 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:07.253685 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:07.254036 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:07.254641 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:07.255493 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:07.255723 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:07.255830 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:07.257869 (+ 2039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.67:40049: BlockingRecv error: recv got EOF from 127.21.143.67:40049 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260619 18:43:07.258142 31234 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 18:43:07 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:07 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:07.890913 31239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:07.885572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.65:44709)
0619 18:43:07.885866 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:07.885882 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:07.885985 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:07.886314 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:07.886316 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:07.886550 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:07.886702 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:07.886708 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:07.887529 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:07.887532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:07.888099 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:07.888106 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:07.888510 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:07.888512 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:07.888592 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:07.889054 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:07.889066 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:07.890798 (+ 1732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":99,"threads_started":1}
W20260619 18:43:07.891068 31240 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:07.885637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:44709 (local address 127.21.143.126:40281)
0619 18:43:07.885945 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:07.885949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:07.885965 (+ 16us) server_negotiation.cc:407] Connection header received
0619 18:43:07.886093 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:07.886097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:07.886159 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:07.886274 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:07.886823 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:07.887427 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:07.888202 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:07.888418 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:07.888493 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:07.890934 (+ 2441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.65:44709: BlockingRecv error: recv got EOF from 127.21.143.65:44709 (error 108)
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":100,"threads_started":1}
Jun 19 18:43:08 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:08 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:08.074975 31241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:08.069307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.66:53853)
0619 18:43:08.069582 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:08.069596 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:08.069691 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:08.069923 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:08.069925 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:08.070098 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:08.070192 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:08.070197 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:08.071025 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:08.071030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:08.071808 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:08.071818 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:08.072305 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:08.072309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:08.072389 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:08.072945 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:08.072957 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:08.074798 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":104,"threads_started":1}
W20260619 18:43:08.075059 31240 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:08.069453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:53853 (local address 127.21.143.126:40281)
0619 18:43:08.069589 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:08.069592 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:08.069663 (+ 71us) server_negotiation.cc:407] Connection header received
0619 18:43:08.069769 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:08.069771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:08.069813 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:08.069886 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:08.070286 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:08.070899 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:08.071931 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:08.072189 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:08.072301 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:08.074913 (+ 2612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.66:53853: BlockingRecv error: recv got EOF from 127.21.143.66:53853 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 18:43:08 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:08 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:08.263741 31242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:08.258751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.67:49671)
0619 18:43:08.259030 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:08.259043 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:08.259163 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:08.259408 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:08.259410 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:08.259577 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:08.259671 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:08.259675 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:08.260362 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:08.260364 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:08.260918 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:08.260924 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:08.261311 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:08.261313 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:08.261374 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:08.261824 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:08.261835 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:08.263611 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":83,"threads_started":1}
W20260619 18:43:08.263916 31240 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:08.258856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:49671 (local address 127.21.143.126:40281)
0619 18:43:08.259049 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:08.259052 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:08.259094 (+ 42us) server_negotiation.cc:407] Connection header received
0619 18:43:08.259246 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:08.259248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:08.259292 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:08.259373 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:08.259766 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:08.260278 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:08.261014 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:08.261219 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:08.261292 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:08.263751 (+ 2459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.67:49671: BlockingRecv error: recv got EOF from 127.21.143.67:49671 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 19 18:43:08 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:08 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:08.897099 31243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:08.891730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.65:56333)
0619 18:43:08.892005 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:08.892023 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:08.892146 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:08.892391 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:08.892393 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:08.892608 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:08.892702 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:08.892707 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:08.893404 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:08.893407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:08.894037 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:08.894044 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:08.894458 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:08.894461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:08.894527 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:08.895100 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:08.895112 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:08.896917 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":64,"threads_started":1}
W20260619 18:43:08.897157 31244 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:08.891874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:56333 (local address 127.21.143.126:40281)
0619 18:43:08.892142 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:08.892145 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:08.892157 (+ 12us) server_negotiation.cc:407] Connection header received
0619 18:43:08.892239 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:08.892241 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:08.892281 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:08.892352 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:08.892796 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:08.893310 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:08.894148 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:08.894361 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:08.894437 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:08.897041 (+ 2604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.65:56333: BlockingRecv error: recv got EOF from 127.21.143.65:56333 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":89,"threads_started":1}
Jun 19 18:43:09 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:09 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:09.081302 31245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:09.075815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.66:55615)
0619 18:43:09.076070 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:09.076084 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:09.076186 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:09.076473 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:09.076476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:09.076672 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:09.076768 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:09.076773 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:09.077490 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:09.077493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:09.078056 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:09.078064 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:09.078498 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:09.078501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:09.078575 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:09.079101 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:09.079115 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:09.081171 (+ 2056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":66,"threads_started":1}
W20260619 18:43:09.081461 31244 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:09.075932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:55615 (local address 127.21.143.126:40281)
0619 18:43:09.076084 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:09.076088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:09.076135 (+ 47us) server_negotiation.cc:407] Connection header received
0619 18:43:09.076263 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:09.076265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:09.076306 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:09.076403 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:09.076869 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:09.077379 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:09.078152 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:09.078409 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:09.078474 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:09.081309 (+ 2835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.66:55615: BlockingRecv error: recv got EOF from 127.21.143.66:55615 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 18:43:09 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:09 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:09.272748 31246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:09.264611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.67:43033)
0619 18:43:09.264897 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:09.264913 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:09.265050 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:09.265375 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:09.265378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:09.265593 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:09.265725 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:09.265735 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:09.266584 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:09.266587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:09.267135 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:09.267142 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:09.267543 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:09.267546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:09.267606 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:09.270884 (+ 3278us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:09.270896 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:09.272609 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":112,"threads_started":1}
W20260619 18:43:09.272853 31244 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:09.264768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:43033 (local address 127.21.143.126:40281)
0619 18:43:09.264916 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:09.264921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:09.264990 (+ 69us) server_negotiation.cc:407] Connection header received
0619 18:43:09.265144 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:09.265146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:09.265243 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:09.265330 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:09.265841 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:09.266488 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:09.267230 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:09.267454 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:09.267526 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:09.272741 (+ 5215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.67:43033: BlockingRecv error: recv got EOF from 127.21.143.67:43033 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 18:43:09 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:09 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:09.903434 31247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:09.897923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.65:53885)
0619 18:43:09.898225 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:09.898238 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:09.898352 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:09.898582 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:09.898585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:09.898749 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:09.898845 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:09.898850 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:09.899740 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:09.899743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:09.900400 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:09.900409 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:09.900825 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:09.900827 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:09.900887 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:09.901439 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:09.901450 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:09.903230 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":128,"threads_started":1}
W20260619 18:43:09.903481 31248 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:09.897975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:53885 (local address 127.21.143.126:40281)
0619 18:43:09.898220 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:09.898223 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:09.898305 (+ 82us) server_negotiation.cc:407] Connection header received
0619 18:43:09.898428 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:09.898430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:09.898472 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:09.898551 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:09.898938 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:09.899661 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:09.900497 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:09.900730 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:09.900809 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:09.903360 (+ 2551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.65:53885: BlockingRecv error: recv got EOF from 127.21.143.65:53885 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":88,"threads_started":1}
Jun 19 18:43:10 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:10 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:10.087235 31249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:10.082062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.66:45609)
0619 18:43:10.082332 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:10.082344 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:10.082438 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:10.082666 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:10.082668 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:10.082849 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:10.082941 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:10.082946 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:10.083704 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:10.083707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:10.084262 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:10.084269 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:10.084677 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:10.084679 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:10.084740 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:10.085228 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:10.085239 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:10.087060 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":103,"threads_started":1}
W20260619 18:43:10.087348 31248 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:10.082116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:45609 (local address 127.21.143.126:40281)
0619 18:43:10.082263 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:10.082266 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:10.082409 (+ 143us) server_negotiation.cc:407] Connection header received
0619 18:43:10.082515 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:10.082517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:10.082560 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:10.082628 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:10.083035 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:10.083610 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:10.084367 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:10.084581 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:10.084658 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:10.087195 (+ 2537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.66:45609: BlockingRecv error: recv got EOF from 127.21.143.66:45609 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 18:43:10 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:10 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:10.279403 31250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:10.273661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.67:33913)
0619 18:43:10.273948 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:10.273962 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:10.274062 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:10.274318 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:10.274320 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:10.274500 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:10.274595 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:10.274600 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:10.275551 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:10.275555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:10.276252 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:10.276261 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:10.276801 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:10.276804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:10.276867 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:10.277418 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:10.277430 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:10.279227 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":69,"threads_started":1}
W20260619 18:43:10.279557 31248 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:10.273820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:33913 (local address 127.21.143.126:40281)
0619 18:43:10.273994 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:10.273997 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:10.274016 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:43:10.274153 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:10.274156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:10.274203 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:10.274274 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:10.274698 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:10.275430 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:10.276365 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:10.276695 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:10.276790 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:10.279411 (+ 2621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.67:33913: BlockingRecv error: recv got EOF from 127.21.143.67:33913 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 19 18:43:10 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:10 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:10.909838 31252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:10.904284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.65:47177)
0619 18:43:10.904585 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:10.904599 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:10.904693 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:10.904924 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:10.904926 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:10.905104 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:10.905203 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:10.905208 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:10.905965 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:10.905968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:10.906594 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:10.906602 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:10.907062 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:10.907065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:10.907133 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:10.907714 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:10.907727 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:10.909670 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":85,"threads_started":1}
W20260619 18:43:10.909950 31251 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:10.904327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:47177 (local address 127.21.143.126:40281)
0619 18:43:10.904643 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:10.904647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:10.904659 (+ 12us) server_negotiation.cc:407] Connection header received
0619 18:43:10.904766 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:10.904768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:10.904811 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:10.904884 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:10.905328 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:10.905871 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:10.906723 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:10.906954 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:10.907051 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:10.909789 (+ 2738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.65:47177: BlockingRecv error: recv got EOF from 127.21.143.65:47177 (error 108)
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":79,"threads_started":1}
Jun 19 18:43:11 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:11 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:11.093678 31253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:11.088120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.66:37349)
0619 18:43:11.088440 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:11.088456 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:11.088580 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:11.088861 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:11.088864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:11.089104 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:11.089217 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:11.089223 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:11.090003 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:11.090009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:11.090574 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:11.090581 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:11.091058 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:11.091062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:11.091146 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:11.091734 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:11.091748 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:11.093512 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":96,"threads_started":1}
W20260619 18:43:11.093741 31251 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:11.088177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:37349 (local address 127.21.143.126:40281)
0619 18:43:11.088323 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:11.088327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:11.088540 (+ 213us) server_negotiation.cc:407] Connection header received
0619 18:43:11.088695 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:11.088700 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:11.088748 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:11.088835 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:11.089340 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:11.089891 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:11.090701 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:11.090927 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:11.091054 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:11.093633 (+ 2579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.66:37349: BlockingRecv error: recv got EOF from 127.21.143.66:37349 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 18:43:11 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:11 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:11.287186 31254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:11.280221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.67:56527)
0619 18:43:11.280537 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:11.280556 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:11.280685 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:11.281000 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:11.281005 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:11.281253 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:11.281391 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:11.281399 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:11.282439 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:11.282444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:11.283425 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:11.283440 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:11.284184 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:11.284193 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:11.284310 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:11.284846 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:11.284873 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:11.287021 (+ 2148us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":102,"threads_started":1}
W20260619 18:43:11.287232 31251 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:11.280351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:56527 (local address 127.21.143.126:40281)
0619 18:43:11.280519 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:11.280524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:11.280623 (+ 99us) server_negotiation.cc:407] Connection header received
0619 18:43:11.280789 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:11.280792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:11.280858 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:11.280964 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:11.281538 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:11.282317 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:11.283590 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:11.284039 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:11.284178 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:11.287132 (+ 2954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.67:56527: BlockingRecv error: recv got EOF from 127.21.143.67:56527 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 19 18:43:11 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:11 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:11.916267 31255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:11.910706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.65:36177)
0619 18:43:11.911013 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:11.911026 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:11.911145 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:11.911514 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:11.911517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:11.911711 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:11.911843 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:11.911850 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:11.912696 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:11.912699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:11.913285 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:11.913293 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:11.913767 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:11.913769 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:11.913831 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:11.914384 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:11.914420 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:11.916158 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":75,"threads_started":1}
W20260619 18:43:11.916441 31256 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:11.910762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:36177 (local address 127.21.143.126:40281)
0619 18:43:11.911028 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:11.911032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:11.911092 (+ 60us) server_negotiation.cc:407] Connection header received
0619 18:43:11.911251 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:11.911257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:11.911354 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:11.911469 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:11.911962 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:11.912576 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:11.913387 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:11.913623 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:11.913740 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:11.916281 (+ 2541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.65:36177: BlockingRecv error: recv got EOF from 127.21.143.65:36177 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":70,"threads_started":1}
Jun 19 18:43:12 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:12 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:12.103178 31257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:12.094546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.66:59201)
0619 18:43:12.094820 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:12.094833 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:12.094919 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:12.095176 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:12.095178 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:12.095378 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:12.095468 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:12.095473 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:12.096279 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:12.096282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:12.097068 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:12.097078 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:12.097477 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:12.097480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:12.097540 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:12.101226 (+ 3686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:12.101237 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:12.102998 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":106,"threads_started":1}
W20260619 18:43:12.103226 31256 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:12.094692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:59201 (local address 127.21.143.126:40281)
0619 18:43:12.094844 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:12.094847 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:12.094900 (+ 53us) server_negotiation.cc:407] Connection header received
0619 18:43:12.095018 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:12.095020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:12.095070 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:12.095141 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:12.095592 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:12.096171 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:12.097170 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:12.097386 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:12.097465 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:12.103123 (+ 5658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.66:59201: BlockingRecv error: recv got EOF from 127.21.143.66:59201 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 18:43:12 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:12 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:12.293388 31258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:12.288041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.67:59011)
0619 18:43:12.288352 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:12.288370 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:12.288494 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:12.288774 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:12.288776 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:12.288983 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:12.289070 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:12.289076 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:12.289854 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:12.289859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:12.290430 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:12.290438 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:12.290920 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:12.290924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:12.291010 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:12.291519 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:12.291530 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:12.293276 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":91,"threads_started":1}
W20260619 18:43:12.293658 31256 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:12.288123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:59011 (local address 127.21.143.126:40281)
0619 18:43:12.288294 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:12.288300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:12.288447 (+ 147us) server_negotiation.cc:407] Connection header received
0619 18:43:12.288599 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:12.288602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:12.288665 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:12.288754 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:12.289192 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:12.289746 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:12.290549 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:12.290811 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:12.290921 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:12.293402 (+ 2481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.67:59011: BlockingRecv error: recv got EOF from 127.21.143.67:59011 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 18:43:12 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:12 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:12.922473 31259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:12.917072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.65:39003)
0619 18:43:12.917336 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:12.917349 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:12.917434 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:12.917662 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:12.917665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:12.917844 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:12.917937 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:12.917942 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:12.918765 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:12.918768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:12.919398 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:12.919410 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:12.919833 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:12.919836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:12.919900 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:12.920497 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:12.920514 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:12.922356 (+ 1842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":106,"threads_started":1}
W20260619 18:43:12.922662 31260 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:12.917167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:39003 (local address 127.21.143.126:40281)
0619 18:43:12.917393 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:12.917397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:12.917409 (+ 12us) server_negotiation.cc:407] Connection header received
0619 18:43:12.917518 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:12.917520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:12.917560 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:12.917644 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:12.918035 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:12.918564 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:12.919507 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:12.919736 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:12.919813 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:12.922491 (+ 2678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.65:39003: BlockingRecv error: recv got EOF from 127.21.143.65:39003 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":91,"threads_started":1}
Jun 19 18:43:13 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:13 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:13.109763 31261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:13.104030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.66:53383)
0619 18:43:13.104283 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:13.104295 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:13.104403 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:13.104629 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:13.104631 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:13.104800 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:13.104889 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:13.104894 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:13.105733 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:13.105736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:13.106304 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:13.106312 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:13.106873 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:13.106877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:13.106961 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:13.107575 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:13.107586 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:13.109653 (+ 2067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":63,"threads_started":1}
W20260619 18:43:13.109923 31260 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:13.104139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:53383 (local address 127.21.143.126:40281)
0619 18:43:13.104299 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:13.104302 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:13.104371 (+ 69us) server_negotiation.cc:407] Connection header received
0619 18:43:13.104479 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:13.104481 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:13.104523 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:13.104588 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:13.105039 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:13.105617 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:13.106413 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:13.106728 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:13.106855 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:13.109770 (+ 2915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.66:53383: BlockingRecv error: recv got EOF from 127.21.143.66:53383 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 18:43:13 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:13 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:13.299124 31262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:13.294055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.67:47773)
0619 18:43:13.294315 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:13.294327 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:13.294435 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:13.294643 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:13.294646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:13.294832 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:13.294946 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:13.294951 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:13.295685 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:13.295688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:13.296238 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:13.296246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:13.296635 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:13.296637 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:13.296710 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:13.297174 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:13.297185 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:13.299012 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":96,"threads_started":1}
W20260619 18:43:13.299343 31260 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:13.294133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:47773 (local address 127.21.143.126:40281)
0619 18:43:13.294273 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:13.294277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:13.294375 (+ 98us) server_negotiation.cc:407] Connection header received
0619 18:43:13.294508 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:13.294510 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:13.294551 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:13.294608 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:13.295044 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:13.295583 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:13.296338 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:13.296545 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:13.296616 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:13.299132 (+ 2516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.67:47773: BlockingRecv error: recv got EOF from 127.21.143.67:47773 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 18:43:13 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:13 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:13.928484 31263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:13.923420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.65:39783)
0619 18:43:13.923762 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:13.923774 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:13.923871 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:13.924101 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:13.924103 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:13.924291 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:13.924383 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:13.924388 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:13.925087 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:13.925089 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:13.925651 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:13.925658 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:13.926053 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:13.926055 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:13.926114 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:13.926648 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:13.926659 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:13.928374 (+ 1715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":94,"threads_started":1}
W20260619 18:43:13.928632 31264 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:13.923458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:39783 (local address 127.21.143.126:40281)
0619 18:43:13.923756 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:13.923759 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:13.923842 (+ 83us) server_negotiation.cc:407] Connection header received
0619 18:43:13.923949 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:13.923952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:13.923994 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:13.924067 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:13.924477 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:13.925003 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:13.925746 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:13.925968 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:13.926037 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:13.928500 (+ 2463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.65:39783: BlockingRecv error: recv got EOF from 127.21.143.65:39783 (error 108)
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":107,"threads_started":1}
Jun 19 18:43:14 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:14 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:14.115983 31265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:14.110592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.66:58593)
0619 18:43:14.110898 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:14.110918 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:14.111029 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:14.111281 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:14.111283 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:14.111524 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:14.111624 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:14.111630 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:14.112368 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:14.112370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:14.112919 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:14.112926 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:14.113336 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:14.113338 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:14.113400 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:14.113904 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:14.113915 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:14.115859 (+ 1944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":110,"threads_started":1}
W20260619 18:43:14.116201 31264 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:14.110659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:58593 (local address 127.21.143.126:40281)
0619 18:43:14.110809 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:14.110813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:14.111007 (+ 194us) server_negotiation.cc:407] Connection header received
0619 18:43:14.111117 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:14.111120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:14.111167 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:14.111229 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:14.111731 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:14.112274 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:14.113016 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:14.113232 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:14.113314 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:14.116043 (+ 2729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.66:58593: BlockingRecv error: recv got EOF from 127.21.143.66:58593 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 18:43:14 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:14 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:14.308637 31266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:14.299960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.67:54643)
0619 18:43:14.300296 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:14.300308 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:14.300548 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:14.300857 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:14.300860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:14.301106 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:14.301195 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:14.301201 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:14.301967 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:14.301973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:14.302656 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:14.302673 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:14.303133 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:14.303136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:14.303220 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:14.306459 (+ 3239us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:14.306474 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:14.308510 (+ 2036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":117,"threads_started":1}
W20260619 18:43:14.309005 31264 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:14.300037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:54643 (local address 127.21.143.126:40281)
0619 18:43:14.300165 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:14.300171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:14.300381 (+ 210us) server_negotiation.cc:407] Connection header received
0619 18:43:14.300654 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:14.300657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:14.300715 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:14.300804 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:14.301318 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:14.301855 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:14.302780 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:14.303022 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:14.303130 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:14.308720 (+ 5590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.67:54643: BlockingRecv error: recv got EOF from 127.21.143.67:54643 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 19 18:43:14 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:14 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:14.934314 31267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:14.929188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.65:59845)
0619 18:43:14.929472 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:14.929486 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:14.929573 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:14.929838 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:14.929840 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:14.930007 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:14.930101 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:14.930106 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:14.930801 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:14.930803 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:14.931376 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:14.931384 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:14.931799 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:14.931802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:14.931873 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:14.932422 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:14.932433 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:14.934203 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":102,"threads_started":1}
W20260619 18:43:14.934543 31268 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:14.929341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:59845 (local address 127.21.143.126:40281)
0619 18:43:14.929545 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:14.929548 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:14.929561 (+ 13us) server_negotiation.cc:407] Connection header received
0619 18:43:14.929657 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:14.929659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:14.929702 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:14.929788 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:14.930193 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:14.930717 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:14.931480 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:14.931711 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:14.931786 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:14.934366 (+ 2580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.65:59845: BlockingRecv error: recv got EOF from 127.21.143.65:59845 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":56,"threads_started":1}
Jun 19 18:43:15 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:15 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:15.122746 31269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:15.116780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.66:51449)
0619 18:43:15.117047 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:15.117065 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:15.117196 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:15.117476 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:15.117479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:15.117715 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:15.117840 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:15.117853 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:15.119047 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:15.119050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:15.119683 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:15.119692 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:15.120215 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:15.120218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:15.120279 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:15.120750 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:15.120761 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:15.122631 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":79,"threads_started":1}
W20260619 18:43:15.122964 31268 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:15.116838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:51449 (local address 127.21.143.126:40281)
0619 18:43:15.117015 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:15.117020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:15.117135 (+ 115us) server_negotiation.cc:407] Connection header received
0619 18:43:15.117304 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:15.117306 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:15.117359 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:15.117451 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:15.117993 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:15.118929 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:15.119795 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:15.120115 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:15.120209 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:15.122833 (+ 2624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.66:51449: BlockingRecv error: recv got EOF from 127.21.143.66:51449 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 19 18:43:15 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:15 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:15.315079 31270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:15.309413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.67:59551)
0619 18:43:15.309668 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:15.309681 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:15.309824 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:15.310091 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:15.310094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:15.310291 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:15.310383 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:15.310389 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:15.311256 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:15.311261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:15.311836 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:15.311844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:15.312316 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:15.312319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:15.312403 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:15.312865 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:15.312876 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:15.314932 (+ 2056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":85,"threads_started":1}
W20260619 18:43:15.315372 31268 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:15.309486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:59551 (local address 127.21.143.126:40281)
0619 18:43:15.309644 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:15.309647 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:15.309757 (+ 110us) server_negotiation.cc:407] Connection header received
0619 18:43:15.309922 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:15.309925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:15.309975 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:15.310069 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:15.310498 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:15.311151 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:15.311961 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:15.312204 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:15.312313 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:15.315107 (+ 2794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.67:59551: BlockingRecv error: recv got EOF from 127.21.143.67:59551 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 18:43:15 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:15 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:15.941030 31271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:15.935171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.65:34961)
0619 18:43:15.935460 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:15.935482 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:15.935570 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:15.935844 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:15.935847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:15.936021 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:15.936142 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:15.936147 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:15.936890 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:15.936893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:15.937612 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:15.937620 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:15.938024 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:15.938026 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:15.938098 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:15.938653 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:15.938672 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:15.940836 (+ 2164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":135,"threads_started":1}
W20260619 18:43:15.941072 31272 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:15.935247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:34961 (local address 127.21.143.126:40281)
0619 18:43:15.935512 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:15.935516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:15.935529 (+ 13us) server_negotiation.cc:407] Connection header received
0619 18:43:15.935669 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:15.935671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:15.935715 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:15.935796 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:15.936261 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:15.936795 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:15.937709 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:15.937929 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:15.938009 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:15.940953 (+ 2944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.65:34961: BlockingRecv error: recv got EOF from 127.21.143.65:34961 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":117,"threads_started":1}
Jun 19 18:43:16 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:16 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:16.128888 31273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:16.123581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.66:38963)
0619 18:43:16.123871 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:16.123894 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:16.124016 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:16.124341 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:16.124348 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:16.124555 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:16.124661 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:16.124667 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:16.125380 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:16.125382 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:16.125932 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:16.125939 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:16.126327 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:16.126329 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:16.126390 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:16.126842 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:16.126853 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:16.128749 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":78,"threads_started":1}
W20260619 18:43:16.129032 31272 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:16.123679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:38963 (local address 127.21.143.126:40281)
0619 18:43:16.123846 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:16.123850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:16.123977 (+ 127us) server_negotiation.cc:407] Connection header received
0619 18:43:16.124117 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:16.124119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:16.124189 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:16.124278 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:16.124766 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:16.125285 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:16.126027 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:16.126234 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:16.126305 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:16.128881 (+ 2576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.66:38963: BlockingRecv error: recv got EOF from 127.21.143.66:38963 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 18:43:16 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:16 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:16.321252 31274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:16.315761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.67:39515)
0619 18:43:16.316029 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:16.316042 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:16.316165 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:16.316403 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:16.316405 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:16.316573 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:16.316670 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:16.316675 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:16.317394 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:16.317396 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:16.317969 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:16.317975 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:16.318369 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:16.318371 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:16.318433 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:16.318901 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:16.318925 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:16.321111 (+ 2186us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":99,"threads_started":1}
W20260619 18:43:16.321388 31272 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:16.315816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:39515 (local address 127.21.143.126:40281)
0619 18:43:16.315935 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:16.315938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:16.316121 (+ 183us) server_negotiation.cc:407] Connection header received
0619 18:43:16.316258 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:16.316261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:16.316304 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:16.316372 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:16.316760 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:16.317287 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:16.318063 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:16.318274 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:16.318350 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:16.321262 (+ 2912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.67:39515: BlockingRecv error: recv got EOF from 127.21.143.67:39515 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 19 18:43:16 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:16 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:16.947093 31275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:16.941869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.65:55119)
0619 18:43:16.942133 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:16.942146 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:16.942226 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:16.942476 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:16.942478 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:16.942658 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:16.942750 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:16.942755 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:16.943538 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:16.943541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:16.944090 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:16.944096 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:16.944485 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:16.944488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:16.944547 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:16.945103 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:16.945114 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:16.946976 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":85,"threads_started":1}
W20260619 18:43:16.947268 31276 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:16.941916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:55119 (local address 127.21.143.126:40281)
0619 18:43:16.942170 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:16.942173 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:16.942188 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:43:16.942313 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:16.942316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:16.942359 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:16.942444 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:16.942844 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:16.943445 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:16.944181 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:16.944395 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:16.944470 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:16.947096 (+ 2626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.65:55119: BlockingRecv error: recv got EOF from 127.21.143.65:55119 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":58,"threads_started":1}
Jun 19 18:43:17 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:17 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:17.135327 31277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:17.129670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.66:43557)
0619 18:43:17.129979 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:17.130002 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:17.130135 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:17.130430 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:17.130433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:17.130636 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:17.130726 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:17.130731 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:17.131557 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:17.131562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:17.132123 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:17.132131 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:17.132650 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:17.132653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:17.132730 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:17.133268 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:17.133279 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:17.135129 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":85,"threads_started":1}
W20260619 18:43:17.135603 31276 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:17.129777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:43557 (local address 127.21.143.126:40281)
0619 18:43:17.129939 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:17.129944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:17.130077 (+ 133us) server_negotiation.cc:407] Connection header received
0619 18:43:17.130253 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:17.130256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:17.130313 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:17.130406 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:17.130840 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:17.131435 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:17.132305 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:17.132538 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:17.132649 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:17.135270 (+ 2621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.66:43557: BlockingRecv error: recv got EOF from 127.21.143.66:43557 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 18:43:17 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:17 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:17.327422 31278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:17.322140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.67:51195)
0619 18:43:17.322432 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:17.322447 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:17.322541 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:17.322784 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:17.322786 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:17.322970 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:17.323067 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:17.323072 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:17.323815 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:17.323817 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:17.324417 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:17.324425 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:17.324856 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:17.324858 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:17.324916 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:17.325419 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:17.325430 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:17.327225 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":116,"threads_started":1}
W20260619 18:43:17.327544 31276 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:17.322272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:51195 (local address 127.21.143.126:40281)
0619 18:43:17.322411 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:17.322415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:17.322514 (+ 99us) server_negotiation.cc:407] Connection header received
0619 18:43:17.322621 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:17.322623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:17.322671 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:17.322740 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:17.323170 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:17.323718 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:17.324535 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:17.324757 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:17.324837 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:17.327425 (+ 2588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.67:51195: BlockingRecv error: recv got EOF from 127.21.143.67:51195 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 18:43:17 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:17 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:17.953118 31279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:17.947881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.65:42461)
0619 18:43:17.948185 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:17.948198 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:17.948301 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:17.948529 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:17.948532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:17.948708 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:17.948802 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:17.948807 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:17.949577 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:17.949580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:17.950156 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:17.950164 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:17.950580 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:17.950582 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:17.950645 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:17.951203 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:17.951215 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:17.953007 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":81,"threads_started":1}
W20260619 18:43:17.953263 31280 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:17.947927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:42461 (local address 127.21.143.126:40281)
0619 18:43:17.948184 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:17.948187 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:17.948249 (+ 62us) server_negotiation.cc:407] Connection header received
0619 18:43:17.948367 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:17.948369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:17.948410 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:17.948484 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:17.948902 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:17.949463 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:17.950268 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:17.950486 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:17.950558 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:17.953129 (+ 2571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.65:42461: BlockingRecv error: recv got EOF from 127.21.143.65:42461 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":66,"threads_started":1}
Jun 19 18:43:18 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:18 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:18.141599 31281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:18.136079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.66:42167)
0619 18:43:18.136370 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:18.136383 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:18.136505 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:18.136808 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:18.136811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:18.137018 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:18.137137 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:18.137144 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:18.137959 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:18.137964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:18.138546 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:18.138554 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:18.139013 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:18.139017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:18.139134 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:18.139659 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:18.139671 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:18.141430 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":81,"threads_started":1}
W20260619 18:43:18.141657 31280 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:18.136142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:42167 (local address 127.21.143.126:40281)
0619 18:43:18.136303 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:18.136308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:18.136450 (+ 142us) server_negotiation.cc:407] Connection header received
0619 18:43:18.136630 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:18.136634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:18.136688 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:18.136777 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:18.137247 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:18.137855 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:18.138663 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:18.138894 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:18.138999 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:18.141550 (+ 2551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.66:42167: BlockingRecv error: recv got EOF from 127.21.143.66:42167 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 18:43:18 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:18 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:18.333757 31282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:18.328373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.67:49385)
0619 18:43:18.328654 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:18.328667 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:18.328768 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:18.328984 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:18.328986 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:18.329168 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:18.329269 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:18.329274 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:18.330048 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:18.330050 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:18.330620 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:18.330627 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:18.331047 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:18.331049 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:18.331108 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:18.331586 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:18.331597 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:18.333579 (+ 1982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":92,"threads_started":1}
W20260619 18:43:18.334007 31280 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:18.328462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:49385 (local address 127.21.143.126:40281)
0619 18:43:18.328606 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:18.328613 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:18.328739 (+ 126us) server_negotiation.cc:407] Connection header received
0619 18:43:18.328835 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:18.328837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:18.328883 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:18.328948 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:18.329419 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:18.329954 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:18.330730 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:18.330952 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:18.331034 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:18.333731 (+ 2697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.67:49385: BlockingRecv error: recv got EOF from 127.21.143.67:49385 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 18:43:18 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:18 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:18.959571 31283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:18.954013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.65:33931)
0619 18:43:18.954274 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:18.954288 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:18.954399 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:18.954636 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:18.954638 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:18.954804 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:18.954895 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:18.954899 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:18.956096 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:18.956103 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:18.956730 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:18.956739 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:18.957142 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:18.957144 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:18.957204 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:18.957767 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:18.957779 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:18.959463 (+ 1684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":86,"threads_started":1}
W20260619 18:43:18.959730 31284 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:18.954067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:33931 (local address 127.21.143.126:40281)
0619 18:43:18.954297 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:18.954300 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:18.954356 (+ 56us) server_negotiation.cc:407] Connection header received
0619 18:43:18.954479 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:18.954481 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:18.954523 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:18.954599 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:18.955018 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:18.955891 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:18.956832 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:18.957050 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:18.957130 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:18.959595 (+ 2465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.65:33931: BlockingRecv error: recv got EOF from 127.21.143.65:33931 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":60,"threads_started":1}
Jun 19 18:43:19 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:19 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:19.148027 31285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:19.142360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.66:58237)
0619 18:43:19.142674 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:19.142687 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:19.142825 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:19.143097 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:19.143099 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:19.143340 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:19.143434 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:19.143441 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:19.144197 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:19.144203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:19.144801 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:19.144810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:19.145270 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:19.145274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:19.145361 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:19.145908 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:19.145921 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:19.147858 (+ 1937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":106,"threads_started":1}
W20260619 18:43:19.148072 31284 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:19.142410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:58237 (local address 127.21.143.126:40281)
0619 18:43:19.142612 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:19.142616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:19.142765 (+ 149us) server_negotiation.cc:407] Connection header received
0619 18:43:19.142924 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:19.142928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:19.142982 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:19.143076 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:19.143553 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:19.144087 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:19.144921 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:19.145141 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:19.145249 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:19.147973 (+ 2724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.66:58237: BlockingRecv error: recv got EOF from 127.21.143.66:58237 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 19 18:43:19 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:19 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:19.343761 31286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:19.334637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.67:49257)
0619 18:43:19.334918 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:19.334932 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:19.335082 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:19.335411 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:19.335413 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:19.335695 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:19.335807 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:19.335813 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:19.336660 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:19.336666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:19.337330 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:19.337339 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:19.337837 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:19.337841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:19.337922 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:19.341824 (+ 3902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:19.341836 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:19.343591 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":92,"threads_started":1}
W20260619 18:43:19.343832 31284 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:19.334700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:49257 (local address 127.21.143.126:40281)
0619 18:43:19.334843 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:19.334847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:19.335023 (+ 176us) server_negotiation.cc:407] Connection header received
0619 18:43:19.335198 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:19.335204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:19.335262 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:19.335393 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:19.335926 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:19.336547 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:19.337456 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:19.337729 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:19.337834 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:19.343722 (+ 5888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.67:49257: BlockingRecv error: recv got EOF from 127.21.143.67:49257 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 18:43:19 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:19 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:19.965654 31287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:19.960399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.65:52591)
0619 18:43:19.960670 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:19.960691 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:19.960782 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:19.961018 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:19.961020 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:19.961195 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:19.961286 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:19.961290 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:19.961962 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:19.961964 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:19.962494 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:19.962501 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:19.962901 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:19.962904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:19.962965 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:19.963471 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:19.963482 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:19.965544 (+ 2062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":74,"threads_started":1}
W20260619 18:43:19.965847 31288 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:19.960529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:52591 (local address 127.21.143.126:40281)
0619 18:43:19.960768 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:19.960772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:19.960784 (+ 12us) server_negotiation.cc:407] Connection header received
0619 18:43:19.960860 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:19.960862 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:19.960901 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:19.960982 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:19.961378 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:19.961880 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:19.962585 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:19.962792 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:19.962861 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:19.965684 (+ 2823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.65:52591: BlockingRecv error: recv got EOF from 127.21.143.65:52591 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":71,"threads_started":1}
Jun 19 18:43:20 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:20 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:20.154198 31289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:20.148765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.66:57331)
0619 18:43:20.149067 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:20.149084 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:20.149230 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:20.149500 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:20.149503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:20.149711 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:20.149795 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:20.149800 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:20.150564 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:20.150569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:20.151144 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:20.151151 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:20.151663 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:20.151666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:20.151752 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:20.152226 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:20.152237 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:20.154078 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":97,"threads_started":1}
W20260619 18:43:20.154322 31288 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:20.148925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:57331 (local address 127.21.143.126:40281)
0619 18:43:20.149069 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:20.149075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:20.149168 (+ 93us) server_negotiation.cc:407] Connection header received
0619 18:43:20.149325 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:20.149329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:20.149381 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:20.149491 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:20.149908 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:20.150449 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:20.151260 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:20.151553 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:20.151661 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:20.154209 (+ 2548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.66:57331: BlockingRecv error: recv got EOF from 127.21.143.66:57331 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 18:43:20 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:20 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:20.349444 31290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:20.344591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.67:33893)
0619 18:43:20.344830 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:20.344843 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:20.344937 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:20.345156 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:20.345159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:20.345334 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:20.345426 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:20.345430 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:20.346119 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:20.346121 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:20.346700 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:20.346708 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:20.347088 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:20.347090 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:20.347149 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:20.347615 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:20.347626 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:20.349308 (+ 1682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":82,"threads_started":1}
W20260619 18:43:20.349519 31288 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:20.344627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:33893 (local address 127.21.143.126:40281)
0619 18:43:20.344757 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:20.344760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:20.344906 (+ 146us) server_negotiation.cc:407] Connection header received
0619 18:43:20.345013 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:20.345015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:20.345060 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:20.345124 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:20.345515 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:20.346034 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:20.346792 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:20.347000 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:20.347070 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:20.349428 (+ 2358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.67:33893: BlockingRecv error: recv got EOF from 127.21.143.67:33893 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 18:43:20 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:20 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:20.971936 31291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:20.966430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.65:37801)
0619 18:43:20.966840 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:20.966857 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:20.966986 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:20.967296 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:20.967299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:20.967587 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:20.967717 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:20.967724 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:20.968430 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:20.968432 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:20.969012 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:20.969019 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:20.969442 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:20.969444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:20.969519 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:20.969998 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:20.970011 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:20.971816 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":116,"threads_started":1}
W20260619 18:43:20.972105 31292 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:20.966621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:37801 (local address 127.21.143.126:40281)
0619 18:43:20.966942 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:20.966949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:20.966966 (+ 17us) server_negotiation.cc:407] Connection header received
0619 18:43:20.967107 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:20.967110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:20.967165 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:20.967267 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:20.967826 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:20.968345 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:20.969114 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:20.969348 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:20.969423 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:20.971932 (+ 2509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.65:37801: BlockingRecv error: recv got EOF from 127.21.143.65:37801 (error 108)
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":103,"threads_started":1}
Jun 19 18:43:21 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:21 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:21.160225 31293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:21.154995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.66:60161)
0619 18:43:21.155259 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:21.155272 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:21.155387 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:21.155684 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:21.155706 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:21.155897 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:21.155991 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:21.155998 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:21.156716 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:21.156719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:21.157267 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:21.157274 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:21.157653 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:21.157655 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:21.157717 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:21.158167 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:21.158177 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:21.160031 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":108,"threads_started":1}
W20260619 18:43:21.160259 31292 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:21.155065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:60161 (local address 127.21.143.126:40281)
0619 18:43:21.155217 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:21.155223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:21.155362 (+ 139us) server_negotiation.cc:407] Connection header received
0619 18:43:21.155467 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:21.155469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:21.155541 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:21.155656 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:21.156095 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:21.156623 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:21.157368 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:21.157564 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:21.157646 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:21.160160 (+ 2514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.66:60161: BlockingRecv error: recv got EOF from 127.21.143.66:60161 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 18:43:21 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:21 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.67@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:21.355226 31294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:21.350167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.67:42601)
0619 18:43:21.350440 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:21.350454 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:21.350563 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:21.350787 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:21.350789 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:21.350964 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:21.351055 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:21.351060 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:21.351855 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:21.351857 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:21.352418 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:21.352426 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:21.352809 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:21.352812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:21.352871 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:21.353332 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:21.353342 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:21.355048 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":73,"threads_started":1}
W20260619 18:43:21.355262 31292 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:21.350308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:42601 (local address 127.21.143.126:40281)
0619 18:43:21.350451 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:21.350454 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:21.350528 (+ 74us) server_negotiation.cc:407] Connection header received
0619 18:43:21.350639 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:21.350641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:21.350686 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:21.350752 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:21.351151 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:21.351742 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:21.352509 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:21.352716 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:21.352787 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:21.355163 (+ 2376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.67:42601: BlockingRecv error: recv got EOF from 127.21.143.67:42601 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 18:43:21 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:21 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.65@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:21.978303 31295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:21.972834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.65:39383)
0619 18:43:21.973147 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:21.973165 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:21.973289 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:21.973576 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:21.973584 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:21.973834 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:21.973969 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:21.973980 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:21.974852 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:21.974855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:21.975527 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:21.975536 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:21.975939 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:21.975941 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:21.976000 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:21.976472 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:21.976483 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:21.978193 (+ 1710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":113,"threads_started":1}
W20260619 18:43:21.978439 31296 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:21.973008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:39383 (local address 127.21.143.126:40281)
0619 18:43:21.973360 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:21.973363 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:21.973375 (+ 12us) server_negotiation.cc:407] Connection header received
0619 18:43:21.973412 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:21.973414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:21.973453 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:21.973524 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:21.974090 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:21.974756 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:21.975622 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:21.975851 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:21.975925 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:21.978312 (+ 2387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.65:39383: BlockingRecv error: recv got EOF from 127.21.143.65:39383 (error 108)
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":70,"threads_started":1}
Jun 19 18:43:22 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:43:22 dist-test-slave-mld5 krb5kdc[30238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.143.66@KRBTEST.COM for oryx/127.21.143.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:43:22.166195 31297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:22.161128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:40281 (local address 127.21.143.66:48495)
0619 18:43:22.161388 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:22.161401 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:22.161491 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:22.161725 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:22.161727 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:22.161915 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:22.162000 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:22.162005 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:22.162779 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:22.162782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:22.163370 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:22.163378 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:22.163814 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:22.163816 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:22.163877 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:22.164337 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:22.164348 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:22.166079 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.143.126:40281: Server oryx/127.21.143.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":89,"threads_started":1}
W20260619 18:43:22.166330 31296 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:22.161189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:48495 (local address 127.21.143.126:40281)
0619 18:43:22.161335 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:22.161339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:22.161466 (+ 127us) server_negotiation.cc:407] Connection header received
0619 18:43:22.161587 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:22.161590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:22.161635 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:22.161713 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:22.162095 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:22.162673 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:22.163483 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:22.163710 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:22.163791 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:22.166210 (+ 2419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.143.66:48495: BlockingRecv error: recv got EOF from 127.21.143.66:48495 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 18:43:22.252365 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 30821
I20260619 18:43:22.261245 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 30973
I20260619 18:43:22.268420 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 31105
I20260619 18:43:22.276724 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 30751
2026-06-19T18:43:22Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18019 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T18:43:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:43:22Z Disabled control of system clock
I20260619 18:43:22.307224 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:33029
--webserver_interface=127.21.143.126
--webserver_port=0
--builtin_ntp_servers=127.21.143.84:33465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:33029
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 18:43:22.418071 31301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:43:22.418344 31301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:43:22.418380 31301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:43:22.422289 31301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:43:22.422348 31301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:43:22.422363 31301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:43:22.422380 31301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:43:22.422396 31301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:43:22.427042 31301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:33465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:33029
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:33029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:43:22.428174 31301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:43:22.429594 31301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:43:22.438048 31306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:22.438102 31307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:22.438048 31309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:22.438272 31301 server_base.cc:1034] running on GCE node
I20260619 18:43:22.439109 31301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:43:22.440266 31301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:43:22.441442 31301 hybrid_clock.cc:648] HybridClock initialized: now 1781894602441406 us; error 50 us; skew 500 ppm
I20260619 18:43:22.443492 31301 webserver.cc:460] Webserver started at http://127.21.143.126:40327/ using document root <none> and password file <none>
I20260619 18:43:22.444078 31301 fs_manager.cc:359] Metadata directory not provided
I20260619 18:43:22.444140 31301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:43:22.444381 31301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:43:22.446440 31301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "3521c45ca7fa409ca71f17d078ff766c"
format_stamp: "Formatted at 2026-06-19 18:43:22 on dist-test-slave-mld5"
server_key: "ab73ce9ef2d7f4edfd4144527ad318b2"
server_key_iv: "d760c68aca394af5d4e57c75e21edca7"
server_key_version: "encryptionkey@0"
I20260619 18:43:22.447002 31301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "3521c45ca7fa409ca71f17d078ff766c"
format_stamp: "Formatted at 2026-06-19 18:43:22 on dist-test-slave-mld5"
server_key: "ab73ce9ef2d7f4edfd4144527ad318b2"
server_key_iv: "d760c68aca394af5d4e57c75e21edca7"
server_key_version: "encryptionkey@0"
I20260619 18:43:22.450681 31301 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260619 18:43:22.453516 31315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:22.454806 31301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 18:43:22.454931 31301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "3521c45ca7fa409ca71f17d078ff766c"
format_stamp: "Formatted at 2026-06-19 18:43:22 on dist-test-slave-mld5"
server_key: "ab73ce9ef2d7f4edfd4144527ad318b2"
server_key_iv: "d760c68aca394af5d4e57c75e21edca7"
server_key_version: "encryptionkey@0"
I20260619 18:43:22.455044 31301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:43:22.474200 31301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:43:22.474982 31301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:43:22.475200 31301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:43:22.482241 31301 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:33029
I20260619 18:43:22.482247 31367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:33029 every 8 connection(s)
I20260619 18:43:22.483213 31301 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:43:22.486454 31368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:43:22.492105 31368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3521c45ca7fa409ca71f17d078ff766c: Bootstrap starting.
I20260619 18:43:22.492995 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 31301
I20260619 18:43:22.493109 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:43:22.493419 22077 external_mini_cluster.cc:1428] Setting key 8159e4b4d8fddec7d76b6e7850f93298
I20260619 18:43:22.495067 31368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3521c45ca7fa409ca71f17d078ff766c: Neither blocks nor log segments found. Creating new log.
I20260619 18:43:22.495858 31368 log.cc:824] T 00000000000000000000000000000000 P 3521c45ca7fa409ca71f17d078ff766c: Log is configured to *not* fsync() on all Append() calls
I20260619 18:43:22.497830 31368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3521c45ca7fa409ca71f17d078ff766c: No bootstrap required, opened a new log
I20260619 18:43:22.502235 31368 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3521c45ca7fa409ca71f17d078ff766c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3521c45ca7fa409ca71f17d078ff766c" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33029 } }
I20260619 18:43:22.502444 31368 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3521c45ca7fa409ca71f17d078ff766c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:43:22.502504 31368 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3521c45ca7fa409ca71f17d078ff766c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3521c45ca7fa409ca71f17d078ff766c, State: Initialized, Role: FOLLOWER
I20260619 18:43:22.502614 31371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:22.494661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59948 (local address 127.21.143.126:33029)
0619 18:43:22.495387 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:22.495416 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:22.495451 (+ 35us) server_negotiation.cc:407] Connection header received
0619 18:43:22.496096 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:22.496115 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:22.496484 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:22.496748 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:22.497066 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:22.498051 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:22.499183 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:22.499788 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:22.499936 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:22.500822 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:22.500826 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:22.500839 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:43:22.501203 (+ 364us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:22.501265 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:43:22.501284 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:22.501358 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:22.501540 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 18:43:22.501824 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":129,"threads_started":1}
I20260619 18:43:22.502978 31368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3521c45ca7fa409ca71f17d078ff766c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3521c45ca7fa409ca71f17d078ff766c" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33029 } }
I20260619 18:43:22.503103 31368 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3521c45ca7fa409ca71f17d078ff766c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:43:22.503185 31368 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3521c45ca7fa409ca71f17d078ff766c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:43:22.503279 31368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3521c45ca7fa409ca71f17d078ff766c [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:43:22.504387 31368 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3521c45ca7fa409ca71f17d078ff766c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3521c45ca7fa409ca71f17d078ff766c" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33029 } }
I20260619 18:43:22.504743 31368 leader_election.cc:302] T 00000000000000000000000000000000 P 3521c45ca7fa409ca71f17d078ff766c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3521c45ca7fa409ca71f17d078ff766c; no voters:
I20260619 18:43:22.505067 31368 leader_election.cc:288] T 00000000000000000000000000000000 P 3521c45ca7fa409ca71f17d078ff766c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:43:22.505162 31373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3521c45ca7fa409ca71f17d078ff766c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:43:22.505426 31373 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3521c45ca7fa409ca71f17d078ff766c [term 1 LEADER]: Becoming Leader. State: Replica: 3521c45ca7fa409ca71f17d078ff766c, State: Running, Role: LEADER
I20260619 18:43:22.505764 31373 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3521c45ca7fa409ca71f17d078ff766c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3521c45ca7fa409ca71f17d078ff766c" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33029 } }
I20260619 18:43:22.506286 31368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3521c45ca7fa409ca71f17d078ff766c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:43:22.507567 31375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3521c45ca7fa409ca71f17d078ff766c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3521c45ca7fa409ca71f17d078ff766c. Latest consensus state: current_term: 1 leader_uuid: "3521c45ca7fa409ca71f17d078ff766c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3521c45ca7fa409ca71f17d078ff766c" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33029 } } }
I20260619 18:43:22.507577 31374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3521c45ca7fa409ca71f17d078ff766c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3521c45ca7fa409ca71f17d078ff766c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3521c45ca7fa409ca71f17d078ff766c" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33029 } } }
I20260619 18:43:22.507714 31375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3521c45ca7fa409ca71f17d078ff766c [sys.catalog]: This master's current role is: LEADER
I20260619 18:43:22.507714 31374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3521c45ca7fa409ca71f17d078ff766c [sys.catalog]: This master's current role is: LEADER
I20260619 18:43:22.508066 31382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:43:22.510674 31382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:43:22.516172 31382 catalog_manager.cc:1347] Generated new cluster ID: 133af408edf840d0bcb05c0a2152ea96
I20260619 18:43:22.516263 31382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:43:22.521602 31382 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:43:22.522799 31382 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:43:22.531958 31382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3521c45ca7fa409ca71f17d078ff766c: Generated new TSK 0
I20260619 18:43:22.532873 31382 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:43:22.542251 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.65:0
--local_ip_for_outbound_sockets=127.21.143.65
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:33029
--builtin_ntp_servers=127.21.143.84:33465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 18:43:22.657497 31392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:43:22.657789 31392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:43:22.657867 31392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:43:22.662387 31392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:43:22.662463 31392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:43:22.662521 31392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.65
I20260619 18:43:22.667611 31392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:33465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:33029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.65
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:43:22.668747 31392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:43:22.670145 31392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:43:22.679843 31397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:22.679848 31400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:22.679843 31398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:22.680197 31392 server_base.cc:1034] running on GCE node
I20260619 18:43:22.680644 31392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:43:22.681318 31392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:43:22.682526 31392 hybrid_clock.cc:648] HybridClock initialized: now 1781894602682515 us; error 30 us; skew 500 ppm
I20260619 18:43:22.684473 31392 webserver.cc:460] Webserver started at http://127.21.143.65:40625/ using document root <none> and password file <none>
I20260619 18:43:22.685070 31392 fs_manager.cc:359] Metadata directory not provided
I20260619 18:43:22.685165 31392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:43:22.685400 31392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:43:22.687613 31392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/data/instance:
uuid: "5546220d73124c7f83d07260b4cc9eb5"
format_stamp: "Formatted at 2026-06-19 18:43:22 on dist-test-slave-mld5"
server_key: "48a845897a75214f4578dfa6bae64dba"
server_key_iv: "4d3c422efe286d368095e9d4223c3909"
server_key_version: "encryptionkey@0"
I20260619 18:43:22.688207 31392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance:
uuid: "5546220d73124c7f83d07260b4cc9eb5"
format_stamp: "Formatted at 2026-06-19 18:43:22 on dist-test-slave-mld5"
server_key: "48a845897a75214f4578dfa6bae64dba"
server_key_iv: "4d3c422efe286d368095e9d4223c3909"
server_key_version: "encryptionkey@0"
I20260619 18:43:22.691891 31392 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:43:22.694320 31406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:22.695629 31392 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:43:22.695794 31392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/wal
uuid: "5546220d73124c7f83d07260b4cc9eb5"
format_stamp: "Formatted at 2026-06-19 18:43:22 on dist-test-slave-mld5"
server_key: "48a845897a75214f4578dfa6bae64dba"
server_key_iv: "4d3c422efe286d368095e9d4223c3909"
server_key_version: "encryptionkey@0"
I20260619 18:43:22.695911 31392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:43:22.708288 31392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:43:22.708959 31392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:43:22.709146 31392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:43:22.709687 31392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:43:22.710601 31392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:43:22.710690 31392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:22.710784 31392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:43:22.710835 31392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:22.719489 31392 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.65:42827
I20260619 18:43:22.719509 31519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.65:42827 every 8 connection(s)
I20260619 18:43:22.720357 31392 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
I20260619 18:43:22.728528 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 31392
I20260619 18:43:22.728644 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance
I20260619 18:43:22.728945 22077 external_mini_cluster.cc:1428] Setting key 62826fa3505f0b656f52f58c90cc6790
I20260619 18:43:22.729558 31371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:22.722156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:46391 (local address 127.21.143.126:33029)
0619 18:43:22.722306 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:22.722310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:22.723420 (+ 1110us) server_negotiation.cc:407] Connection header received
0619 18:43:22.724745 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:22.724747 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:22.724795 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:22.724869 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:22.725711 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:22.726244 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:22.727627 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:22.727865 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:22.727948 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:22.729053 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:22.729057 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:22.729060 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:43:22.729074 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:22.729096 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:43:22.729106 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:22.729196 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:22.729373 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 18:43:22.729444 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 18:43:22.730530 31522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:22.722598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33029 (local address 127.21.143.65:46391)
0619 18:43:22.723231 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:22.723271 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:22.724552 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:22.725028 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:22.725034 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:22.725116 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:22.725586 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:22.725598 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:22.726352 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:22.726354 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:22.727523 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:22.727532 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:22.727996 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:22.727999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:22.728132 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:22.728786 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:43:22.728788 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:22.728828 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:43:22.728889 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:43:22.729208 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:43:22.729214 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:43:22.729236 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 18:43:22.729337 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 18:43:22.729653 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":105,"threads_started":1}
I20260619 18:43:22.730959 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.66:0
--local_ip_for_outbound_sockets=127.21.143.66
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:33029
--builtin_ntp_servers=127.21.143.84:33465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 18:43:22.731807 31520 heartbeater.cc:347] Connected to a master server at 127.21.143.126:33029
I20260619 18:43:22.732038 31520 heartbeater.cc:464] Registering TS with master...
I20260619 18:43:22.732591 31520 heartbeater.cc:511] Master 127.21.143.126:33029 requested a full tablet report, sending...
I20260619 18:43:22.734061 31332 ts_manager.cc:194] Registered new tserver with Master: 5546220d73124c7f83d07260b4cc9eb5 (127.21.143.65:42827)
I20260619 18:43:22.735183 31332 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.143.65:46391
W20260619 18:43:22.841970 31523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:43:22.842253 31523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:43:22.842298 31523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:43:22.846625 31523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:43:22.846683 31523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:43:22.846736 31523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.66
I20260619 18:43:22.851586 31523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:33465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:33029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.66
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:43:22.852756 31523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:43:22.854233 31523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:43:22.864096 31528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:22.864152 31531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:22.864096 31529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:22.864425 31523 server_base.cc:1034] running on GCE node
I20260619 18:43:22.864837 31523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:43:22.865432 31523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:43:22.866580 31523 hybrid_clock.cc:648] HybridClock initialized: now 1781894602866549 us; error 43 us; skew 500 ppm
I20260619 18:43:22.868410 31523 webserver.cc:460] Webserver started at http://127.21.143.66:38811/ using document root <none> and password file <none>
I20260619 18:43:22.868957 31523 fs_manager.cc:359] Metadata directory not provided
I20260619 18:43:22.869015 31523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:43:22.869238 31523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:43:22.871212 31523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/data/instance:
uuid: "74a0f9c734074bd09ef8f34fd0864bbe"
format_stamp: "Formatted at 2026-06-19 18:43:22 on dist-test-slave-mld5"
server_key: "01466e5f55e923bdc382956202e09cd9"
server_key_iv: "fda2e812663613c60b9a1147a36adc28"
server_key_version: "encryptionkey@0"
I20260619 18:43:22.871838 31523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance:
uuid: "74a0f9c734074bd09ef8f34fd0864bbe"
format_stamp: "Formatted at 2026-06-19 18:43:22 on dist-test-slave-mld5"
server_key: "01466e5f55e923bdc382956202e09cd9"
server_key_iv: "fda2e812663613c60b9a1147a36adc28"
server_key_version: "encryptionkey@0"
I20260619 18:43:22.875280 31523 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:43:22.877813 31537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:22.878986 31523 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 18:43:22.879097 31523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/wal
uuid: "74a0f9c734074bd09ef8f34fd0864bbe"
format_stamp: "Formatted at 2026-06-19 18:43:22 on dist-test-slave-mld5"
server_key: "01466e5f55e923bdc382956202e09cd9"
server_key_iv: "fda2e812663613c60b9a1147a36adc28"
server_key_version: "encryptionkey@0"
I20260619 18:43:22.879194 31523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:43:22.902660 31523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:43:22.903532 31523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:43:22.903750 31523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:43:22.904392 31523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:43:22.905380 31523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:43:22.905432 31523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:22.905504 31523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:43:22.905542 31523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:22.915022 31523 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.66:45679
I20260619 18:43:22.915046 31650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.66:45679 every 8 connection(s)
I20260619 18:43:22.916054 31523 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
I20260619 18:43:22.917526 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 31523
I20260619 18:43:22.917662 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance
I20260619 18:43:22.918020 22077 external_mini_cluster.cc:1428] Setting key 2b6c44757fc30997e9a8bf4828cab6f3
I20260619 18:43:22.919997 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.67:0
--local_ip_for_outbound_sockets=127.21.143.67
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:33029
--builtin_ntp_servers=127.21.143.84:33465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 18:43:22.926832 31371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:22.917795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:43011 (local address 127.21.143.126:33029)
0619 18:43:22.917949 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:22.917954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:22.919335 (+ 1381us) server_negotiation.cc:407] Connection header received
0619 18:43:22.920887 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:22.920893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:22.920955 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:22.921045 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:22.922276 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:22.922821 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:22.924701 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:22.924923 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:22.925004 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:22.926349 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:22.926353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:22.926355 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:43:22.926367 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:22.926385 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:43:22.926393 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:22.926450 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:22.926646 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 18:43:22.926701 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 18:43:22.928298 31653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:22.918357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33029 (local address 127.21.143.66:43011)
0619 18:43:22.919142 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:22.919197 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:22.920660 (+ 1463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:22.921234 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:22.921247 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:22.921387 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:22.922110 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:22.922129 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:22.922962 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:22.922968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:22.924575 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:22.924590 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:22.925086 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:22.925092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:22.925350 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:22.926019 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:43:22.926023 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:22.926087 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:43:22.926183 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:43:22.926480 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:43:22.926492 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:43:22.926499 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 18:43:22.926648 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 18:43:22.927093 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":144,"threads_started":1}
I20260619 18:43:22.929518 31651 heartbeater.cc:347] Connected to a master server at 127.21.143.126:33029
I20260619 18:43:22.929787 31651 heartbeater.cc:464] Registering TS with master...
I20260619 18:43:22.930474 31651 heartbeater.cc:511] Master 127.21.143.126:33029 requested a full tablet report, sending...
I20260619 18:43:22.931655 31332 ts_manager.cc:194] Registered new tserver with Master: 74a0f9c734074bd09ef8f34fd0864bbe (127.21.143.66:45679)
I20260619 18:43:22.932231 31332 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.143.66:43011
W20260619 18:43:23.031522 31654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:43:23.031848 31654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:43:23.031922 31654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:43:23.035917 31654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:43:23.035981 31654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:43:23.036062 31654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.67
I20260619 18:43:23.040781 31654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:33465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:33029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.67
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:43:23.041821 31654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:43:23.043148 31654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:43:23.052475 31660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:23.052613 31654 server_base.cc:1034] running on GCE node
W20260619 18:43:23.052477 31662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:23.052474 31659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:23.053167 31654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:43:23.053800 31654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:43:23.054983 31654 hybrid_clock.cc:648] HybridClock initialized: now 1781894603054965 us; error 29 us; skew 500 ppm
I20260619 18:43:23.056810 31654 webserver.cc:460] Webserver started at http://127.21.143.67:41431/ using document root <none> and password file <none>
I20260619 18:43:23.057380 31654 fs_manager.cc:359] Metadata directory not provided
I20260619 18:43:23.057469 31654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:43:23.057693 31654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:43:23.059993 31654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/data/instance:
uuid: "f8e00f5a5cfe4f189698518880034049"
format_stamp: "Formatted at 2026-06-19 18:43:23 on dist-test-slave-mld5"
server_key: "f41ca8c08d64592ab52f531aa22278ef"
server_key_iv: "752415e23b2894044eb79a4a64749355"
server_key_version: "encryptionkey@0"
I20260619 18:43:23.060575 31654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance:
uuid: "f8e00f5a5cfe4f189698518880034049"
format_stamp: "Formatted at 2026-06-19 18:43:23 on dist-test-slave-mld5"
server_key: "f41ca8c08d64592ab52f531aa22278ef"
server_key_iv: "752415e23b2894044eb79a4a64749355"
server_key_version: "encryptionkey@0"
I20260619 18:43:23.064205 31654 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:43:23.066668 31668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:23.067981 31654 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:43:23.068113 31654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/wal
uuid: "f8e00f5a5cfe4f189698518880034049"
format_stamp: "Formatted at 2026-06-19 18:43:23 on dist-test-slave-mld5"
server_key: "f41ca8c08d64592ab52f531aa22278ef"
server_key_iv: "752415e23b2894044eb79a4a64749355"
server_key_version: "encryptionkey@0"
I20260619 18:43:23.068256 31654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:43:23.083988 31654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:43:23.084645 31654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:43:23.084847 31654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:43:23.085446 31654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:43:23.086351 31654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:43:23.086421 31654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:23.086489 31654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:43:23.086537 31654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:23.094821 31654 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.67:41529
I20260619 18:43:23.094843 31781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.67:41529 every 8 connection(s)
I20260619 18:43:23.095732 31654 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
I20260619 18:43:23.096379 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 31654
I20260619 18:43:23.096494 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance
I20260619 18:43:23.096829 22077 external_mini_cluster.cc:1428] Setting key de3682eaa74e73009f057930880852c5
I20260619 18:43:23.104735 31371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:23.097609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:34787 (local address 127.21.143.126:33029)
0619 18:43:23.097735 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:23.097739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:23.098996 (+ 1257us) server_negotiation.cc:407] Connection header received
0619 18:43:23.100396 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:23.100399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:23.100443 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:23.100514 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:23.101319 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:23.101862 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:23.102958 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:23.103161 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:23.103235 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:23.104351 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:23.104354 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:23.104356 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:43:23.104367 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:23.104385 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:43:23.104392 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:23.104448 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:23.104580 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 18:43:23.104629 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 18:43:23.105553 31784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:23.098085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33029 (local address 127.21.143.67:34787)
0619 18:43:23.098826 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:23.098883 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:23.100218 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:23.100641 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:23.100648 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:23.100734 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:23.101210 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:23.101224 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:23.101970 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:23.101973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:23.102863 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:23.102872 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:23.103278 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:23.103281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:23.103488 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:23.104128 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:43:23.104129 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:23.104168 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:43:23.104226 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:43:23.104463 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:43:23.104469 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:43:23.104473 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 18:43:23.104561 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 18:43:23.104803 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":109,"threads_started":1}
I20260619 18:43:23.106767 31782 heartbeater.cc:347] Connected to a master server at 127.21.143.126:33029
I20260619 18:43:23.107071 31782 heartbeater.cc:464] Registering TS with master...
I20260619 18:43:23.107705 31782 heartbeater.cc:511] Master 127.21.143.126:33029 requested a full tablet report, sending...
I20260619 18:43:23.108772 31332 ts_manager.cc:194] Registered new tserver with Master: f8e00f5a5cfe4f189698518880034049 (127.21.143.67:41529)
I20260619 18:43:23.109336 31332 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.143.67:34787
I20260619 18:43:23.110739 22077 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 18:43:23.114554 31371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:23.112633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59964 (local address 127.21.143.126:33029)
0619 18:43:23.112811 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:23.112816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:23.113946 (+ 1130us) server_negotiation.cc:407] Connection header received
0619 18:43:23.114073 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:23.114076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:23.114117 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:23.114206 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:23.114443 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59964: BlockingRecv error: recv got EOF from 127.0.0.1:59964 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 18:43:23.114805 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 31392
I20260619 18:43:23.120894 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 31523
I20260619 18:43:23.126291 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 31654
I20260619 18:43:23.131729 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 31301
2026-06-19T18:43:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (853 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T18:43:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:43:23Z Disabled control of system clock
I20260619 18:43:23.160859 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:38339
--webserver_interface=127.21.143.126
--webserver_port=0
--builtin_ntp_servers=127.21.143.84:37347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:38339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 18:43:23.271025 31793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:43:23.271318 31793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:43:23.271378 31793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:43:23.275305 31793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:43:23.275388 31793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:43:23.275413 31793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:43:23.275430 31793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:43:23.275445 31793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:43:23.280051 31793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:37347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:38339
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:38339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:43:23.281020 31793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:43:23.282328 31793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:43:23.290436 31798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:23.290443 31801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:23.290459 31799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:23.290663 31793 server_base.cc:1034] running on GCE node
I20260619 18:43:23.291389 31793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:43:23.292435 31793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:43:23.293634 31793 hybrid_clock.cc:648] HybridClock initialized: now 1781894603293620 us; error 25 us; skew 500 ppm
I20260619 18:43:23.295493 31793 webserver.cc:460] Webserver started at http://127.21.143.126:35457/ using document root <none> and password file <none>
I20260619 18:43:23.296047 31793 fs_manager.cc:359] Metadata directory not provided
I20260619 18:43:23.296137 31793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:43:23.296365 31793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:43:23.298310 31793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "e772e6fa82224b41ad6138234753bf21"
format_stamp: "Formatted at 2026-06-19 18:43:23 on dist-test-slave-mld5"
server_key: "ff8b72e2d96db75b7a61be85b2aafb33"
server_key_iv: "3dbb6f70e4612e1f0ed62453f0c9b963"
server_key_version: "encryptionkey@0"
I20260619 18:43:23.298893 31793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "e772e6fa82224b41ad6138234753bf21"
format_stamp: "Formatted at 2026-06-19 18:43:23 on dist-test-slave-mld5"
server_key: "ff8b72e2d96db75b7a61be85b2aafb33"
server_key_iv: "3dbb6f70e4612e1f0ed62453f0c9b963"
server_key_version: "encryptionkey@0"
I20260619 18:43:23.302327 31793 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 18:43:23.304625 31807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:23.305749 31793 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:43:23.305881 31793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "e772e6fa82224b41ad6138234753bf21"
format_stamp: "Formatted at 2026-06-19 18:43:23 on dist-test-slave-mld5"
server_key: "ff8b72e2d96db75b7a61be85b2aafb33"
server_key_iv: "3dbb6f70e4612e1f0ed62453f0c9b963"
server_key_version: "encryptionkey@0"
I20260619 18:43:23.305984 31793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:43:23.310177 31793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:43:23.310755 31793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:43:23.310930 31793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:43:23.317473 31793 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:38339
I20260619 18:43:23.317492 31859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:38339 every 8 connection(s)
I20260619 18:43:23.318323 31793 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:43:23.321412 31860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:43:23.326289 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 31793
I20260619 18:43:23.326434 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:43:23.326711 22077 external_mini_cluster.cc:1428] Setting key d5a158c8f3479d71504b94af9880d119
I20260619 18:43:23.326735 31860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e772e6fa82224b41ad6138234753bf21: Bootstrap starting.
I20260619 18:43:23.329203 31860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e772e6fa82224b41ad6138234753bf21: Neither blocks nor log segments found. Creating new log.
I20260619 18:43:23.329950 31860 log.cc:824] T 00000000000000000000000000000000 P e772e6fa82224b41ad6138234753bf21: Log is configured to *not* fsync() on all Append() calls
I20260619 18:43:23.332181 31860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e772e6fa82224b41ad6138234753bf21: No bootstrap required, opened a new log
I20260619 18:43:23.332561 31863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:23.328082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33878 (local address 127.21.143.126:38339)
0619 18:43:23.328724 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:23.328738 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:23.328770 (+ 32us) server_negotiation.cc:407] Connection header received
0619 18:43:23.329442 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:23.329459 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:23.329826 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:23.330158 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:23.331056 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:23.331061 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:23.331076 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:43:23.331087 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:23.331151 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:43:23.331172 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:23.331316 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:23.331447 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 18:43:23.331688 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":132,"threads_started":1}
I20260619 18:43:23.336115 31860 raft_consensus.cc:348] T 00000000000000000000000000000000 P e772e6fa82224b41ad6138234753bf21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e772e6fa82224b41ad6138234753bf21" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 38339 } }
I20260619 18:43:23.336320 31860 raft_consensus.cc:374] T 00000000000000000000000000000000 P e772e6fa82224b41ad6138234753bf21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:43:23.336354 31860 raft_consensus.cc:729] T 00000000000000000000000000000000 P e772e6fa82224b41ad6138234753bf21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e772e6fa82224b41ad6138234753bf21, State: Initialized, Role: FOLLOWER
I20260619 18:43:23.336694 31860 consensus_queue.cc:260] T 00000000000000000000000000000000 P e772e6fa82224b41ad6138234753bf21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e772e6fa82224b41ad6138234753bf21" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 38339 } }
I20260619 18:43:23.336799 31860 raft_consensus.cc:388] T 00000000000000000000000000000000 P e772e6fa82224b41ad6138234753bf21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:43:23.336835 31860 raft_consensus.cc:482] T 00000000000000000000000000000000 P e772e6fa82224b41ad6138234753bf21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:43:23.336903 31860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e772e6fa82224b41ad6138234753bf21 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:43:23.337726 31860 raft_consensus.cc:504] T 00000000000000000000000000000000 P e772e6fa82224b41ad6138234753bf21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e772e6fa82224b41ad6138234753bf21" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 38339 } }
I20260619 18:43:23.338017 31860 leader_election.cc:302] T 00000000000000000000000000000000 P e772e6fa82224b41ad6138234753bf21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e772e6fa82224b41ad6138234753bf21; no voters:
I20260619 18:43:23.338249 31860 leader_election.cc:288] T 00000000000000000000000000000000 P e772e6fa82224b41ad6138234753bf21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:43:23.338361 31865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e772e6fa82224b41ad6138234753bf21 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:43:23.338615 31865 raft_consensus.cc:686] T 00000000000000000000000000000000 P e772e6fa82224b41ad6138234753bf21 [term 1 LEADER]: Becoming Leader. State: Replica: e772e6fa82224b41ad6138234753bf21, State: Running, Role: LEADER
I20260619 18:43:23.338960 31865 consensus_queue.cc:237] T 00000000000000000000000000000000 P e772e6fa82224b41ad6138234753bf21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e772e6fa82224b41ad6138234753bf21" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 38339 } }
I20260619 18:43:23.339481 31860 sys_catalog.cc:564] T 00000000000000000000000000000000 P e772e6fa82224b41ad6138234753bf21 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:43:23.340557 31867 sys_catalog.cc:455] T 00000000000000000000000000000000 P e772e6fa82224b41ad6138234753bf21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e772e6fa82224b41ad6138234753bf21. Latest consensus state: current_term: 1 leader_uuid: "e772e6fa82224b41ad6138234753bf21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e772e6fa82224b41ad6138234753bf21" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 38339 } } }
I20260619 18:43:23.340600 31866 sys_catalog.cc:455] T 00000000000000000000000000000000 P e772e6fa82224b41ad6138234753bf21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e772e6fa82224b41ad6138234753bf21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e772e6fa82224b41ad6138234753bf21" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 38339 } } }
I20260619 18:43:23.340668 31867 sys_catalog.cc:458] T 00000000000000000000000000000000 P e772e6fa82224b41ad6138234753bf21 [sys.catalog]: This master's current role is: LEADER
I20260619 18:43:23.340698 31866 sys_catalog.cc:458] T 00000000000000000000000000000000 P e772e6fa82224b41ad6138234753bf21 [sys.catalog]: This master's current role is: LEADER
I20260619 18:43:23.341022 31874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:43:23.343995 31874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:43:23.349614 31874 catalog_manager.cc:1347] Generated new cluster ID: 8b4cb638f366495b811d4a1a17a8628e
I20260619 18:43:23.349689 31874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:43:23.362301 31874 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:43:23.362609 31874 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:43:23.369948 31874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e772e6fa82224b41ad6138234753bf21: Generated new TSK 0
I20260619 18:43:23.370625 31874 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:43:23.376386 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.65:0
--local_ip_for_outbound_sockets=127.21.143.65
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:38339
--builtin_ntp_servers=127.21.143.84:37347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 18:43:23.491075 31884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:43:23.491361 31884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:43:23.491420 31884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:43:23.495198 31884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:43:23.495236 31884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:43:23.495282 31884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.65
I20260619 18:43:23.500138 31884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:37347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:38339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.65
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:43:23.501094 31884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:43:23.502363 31884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:43:23.511443 31889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:23.511415 31890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:23.511451 31892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:23.511729 31884 server_base.cc:1034] running on GCE node
I20260619 18:43:23.512147 31884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:43:23.512712 31884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:43:23.513906 31884 hybrid_clock.cc:648] HybridClock initialized: now 1781894603513893 us; error 25 us; skew 500 ppm
I20260619 18:43:23.515652 31884 webserver.cc:460] Webserver started at http://127.21.143.65:44345/ using document root <none> and password file <none>
I20260619 18:43:23.516199 31884 fs_manager.cc:359] Metadata directory not provided
I20260619 18:43:23.516286 31884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:43:23.516506 31884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:43:23.518445 31884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/data/instance:
uuid: "70a67f29fd19416aa97ea15ff3226985"
format_stamp: "Formatted at 2026-06-19 18:43:23 on dist-test-slave-mld5"
server_key: "0b9e487b0e474260f70027a249e57ea9"
server_key_iv: "ea7ec069888dfafe5d068bc3a0ccfbb1"
server_key_version: "encryptionkey@0"
I20260619 18:43:23.518981 31884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance:
uuid: "70a67f29fd19416aa97ea15ff3226985"
format_stamp: "Formatted at 2026-06-19 18:43:23 on dist-test-slave-mld5"
server_key: "0b9e487b0e474260f70027a249e57ea9"
server_key_iv: "ea7ec069888dfafe5d068bc3a0ccfbb1"
server_key_version: "encryptionkey@0"
I20260619 18:43:23.522511 31884 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.001s
I20260619 18:43:23.524870 31898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:23.525974 31884 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:43:23.526101 31884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/wal
uuid: "70a67f29fd19416aa97ea15ff3226985"
format_stamp: "Formatted at 2026-06-19 18:43:23 on dist-test-slave-mld5"
server_key: "0b9e487b0e474260f70027a249e57ea9"
server_key_iv: "ea7ec069888dfafe5d068bc3a0ccfbb1"
server_key_version: "encryptionkey@0"
I20260619 18:43:23.526207 31884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:43:23.531751 31884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:43:23.532397 31884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:43:23.532577 31884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:43:23.533133 31884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:43:23.534000 31884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:43:23.534077 31884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:23.534142 31884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:43:23.534190 31884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:23.542807 31884 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.65:38847
I20260619 18:43:23.542874 32011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.65:38847 every 8 connection(s)
I20260619 18:43:23.543722 31884 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
I20260619 18:43:23.549276 31863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:23.545427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:37619 (local address 127.21.143.126:38339)
0619 18:43:23.545566 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:23.545569 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:23.546510 (+ 941us) server_negotiation.cc:407] Connection header received
0619 18:43:23.547827 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:23.547829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:23.547865 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:23.547941 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:23.548913 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:23.548916 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:23.548918 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:43:23.548929 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:23.548947 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:43:23.548954 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:23.549024 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:23.549139 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 18:43:23.549174 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 18:43:23.550056 32014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:23.545785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:38339 (local address 127.21.143.65:37619)
0619 18:43:23.546396 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:23.546432 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:23.547648 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:23.548084 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:23.548090 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:23.548173 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:23.548664 (+ 491us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:43:23.548666 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:23.548702 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:43:23.548764 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:43:23.549026 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:43:23.549033 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:43:23.549036 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 18:43:23.549157 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 18:43:23.549333 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":104,"threads_started":1}
I20260619 18:43:23.551160 32012 heartbeater.cc:347] Connected to a master server at 127.21.143.126:38339
I20260619 18:43:23.551412 32012 heartbeater.cc:464] Registering TS with master...
I20260619 18:43:23.551793 32012 heartbeater.cc:511] Master 127.21.143.126:38339 requested a full tablet report, sending...
I20260619 18:43:23.552623 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 31884
I20260619 18:43:23.552747 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance
I20260619 18:43:23.553035 22077 external_mini_cluster.cc:1428] Setting key 21b46251246d684add2a0d8863cf5483
I20260619 18:43:23.553372 31824 ts_manager.cc:194] Registered new tserver with Master: 70a67f29fd19416aa97ea15ff3226985 (127.21.143.65:38847)
I20260619 18:43:23.554912 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.66:0
--local_ip_for_outbound_sockets=127.21.143.66
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:38339
--builtin_ntp_servers=127.21.143.84:37347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 18:43:23.662428 32015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:43:23.662671 32015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:43:23.662734 32015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:43:23.666688 32015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:43:23.666759 32015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:43:23.666849 32015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.66
I20260619 18:43:23.671535 32015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:37347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:38339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.66
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:43:23.672467 32015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:43:23.673719 32015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:43:23.682679 32023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:23.682698 32020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:23.682654 32021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:23.682807 32015 server_base.cc:1034] running on GCE node
I20260619 18:43:23.683238 32015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:43:23.683887 32015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:43:23.685086 32015 hybrid_clock.cc:648] HybridClock initialized: now 1781894603685077 us; error 28 us; skew 500 ppm
I20260619 18:43:23.686780 32015 webserver.cc:460] Webserver started at http://127.21.143.66:43429/ using document root <none> and password file <none>
I20260619 18:43:23.687495 32015 fs_manager.cc:359] Metadata directory not provided
I20260619 18:43:23.687585 32015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:43:23.687780 32015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:43:23.689677 32015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/data/instance:
uuid: "d44b59484ab44e3baefa308783e09147"
format_stamp: "Formatted at 2026-06-19 18:43:23 on dist-test-slave-mld5"
server_key: "cb93ce16a1e27b3dc6cabe99ef19b1ab"
server_key_iv: "1a414facac7f29b5dabe22c82d10f565"
server_key_version: "encryptionkey@0"
I20260619 18:43:23.690266 32015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance:
uuid: "d44b59484ab44e3baefa308783e09147"
format_stamp: "Formatted at 2026-06-19 18:43:23 on dist-test-slave-mld5"
server_key: "cb93ce16a1e27b3dc6cabe99ef19b1ab"
server_key_iv: "1a414facac7f29b5dabe22c82d10f565"
server_key_version: "encryptionkey@0"
I20260619 18:43:23.693743 32015 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260619 18:43:23.696049 32029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:23.697129 32015 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260619 18:43:23.697254 32015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/wal
uuid: "d44b59484ab44e3baefa308783e09147"
format_stamp: "Formatted at 2026-06-19 18:43:23 on dist-test-slave-mld5"
server_key: "cb93ce16a1e27b3dc6cabe99ef19b1ab"
server_key_iv: "1a414facac7f29b5dabe22c82d10f565"
server_key_version: "encryptionkey@0"
I20260619 18:43:23.697359 32015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:43:23.701655 32015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:43:23.702215 32015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:43:23.702406 32015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:43:23.702934 32015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:43:23.703837 32015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:43:23.703912 32015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:23.703984 32015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:43:23.704018 32015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:23.712971 32015 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.66:41481
I20260619 18:43:23.712989 32142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.66:41481 every 8 connection(s)
I20260619 18:43:23.713801 32015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
I20260619 18:43:23.719724 31863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:23.715526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:37597 (local address 127.21.143.126:38339)
0619 18:43:23.715658 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:23.715662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:23.716695 (+ 1033us) server_negotiation.cc:407] Connection header received
0619 18:43:23.717973 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:23.717977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:23.718020 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:23.718124 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:23.719221 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:23.719224 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:23.719227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:43:23.719239 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:23.719262 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:43:23.719277 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:23.719413 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:23.719548 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 18:43:23.719593 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 18:43:23.720513 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 32015
I20260619 18:43:23.720645 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance
I20260619 18:43:23.720638 32145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:23.715948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:38339 (local address 127.21.143.66:37597)
0619 18:43:23.716560 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:23.716598 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:23.717767 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:23.718281 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:23.718288 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:23.718395 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:23.718966 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:43:23.718968 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:23.719007 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:43:23.719073 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:43:23.719399 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:43:23.719407 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:43:23.719413 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 18:43:23.719591 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 18:43:23.719818 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":106,"threads_started":1}
I20260619 18:43:23.721033 22077 external_mini_cluster.cc:1428] Setting key e1b9e43c8bc85117ece094b3c5339b81
I20260619 18:43:23.721798 32143 heartbeater.cc:347] Connected to a master server at 127.21.143.126:38339
I20260619 18:43:23.722012 32143 heartbeater.cc:464] Registering TS with master...
I20260619 18:43:23.722417 32143 heartbeater.cc:511] Master 127.21.143.126:38339 requested a full tablet report, sending...
I20260619 18:43:23.723280 31824 ts_manager.cc:194] Registered new tserver with Master: d44b59484ab44e3baefa308783e09147 (127.21.143.66:41481)
I20260619 18:43:23.723826 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.67:0
--local_ip_for_outbound_sockets=127.21.143.67
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:38339
--builtin_ntp_servers=127.21.143.84:37347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 18:43:23.829931 32146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:43:23.830163 32146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:43:23.830201 32146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:43:23.834198 32146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:43:23.834246 32146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:43:23.834298 32146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.67
I20260619 18:43:23.838779 32146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:37347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:38339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.67
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:43:23.839797 32146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:43:23.841264 32146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:43:23.850616 32152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:23.850612 32154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:23.850703 32146 server_base.cc:1034] running on GCE node
W20260619 18:43:23.850625 32151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:23.851248 32146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:43:23.851832 32146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:43:23.852993 32146 hybrid_clock.cc:648] HybridClock initialized: now 1781894603852977 us; error 24 us; skew 500 ppm
I20260619 18:43:23.854626 32146 webserver.cc:460] Webserver started at http://127.21.143.67:36423/ using document root <none> and password file <none>
I20260619 18:43:23.855181 32146 fs_manager.cc:359] Metadata directory not provided
I20260619 18:43:23.855239 32146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:43:23.855512 32146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:43:23.857484 32146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/data/instance:
uuid: "01bcc5bc6ce54eb19166ba129a28be95"
format_stamp: "Formatted at 2026-06-19 18:43:23 on dist-test-slave-mld5"
server_key: "f01533bee745dc4e3fb169cde239b38f"
server_key_iv: "f19db04914b7076fa1da356ac90ecf3f"
server_key_version: "encryptionkey@0"
I20260619 18:43:23.858031 32146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance:
uuid: "01bcc5bc6ce54eb19166ba129a28be95"
format_stamp: "Formatted at 2026-06-19 18:43:23 on dist-test-slave-mld5"
server_key: "f01533bee745dc4e3fb169cde239b38f"
server_key_iv: "f19db04914b7076fa1da356ac90ecf3f"
server_key_version: "encryptionkey@0"
I20260619 18:43:23.861526 32146 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260619 18:43:23.863958 32160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:23.865079 32146 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:43:23.865200 32146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/wal
uuid: "01bcc5bc6ce54eb19166ba129a28be95"
format_stamp: "Formatted at 2026-06-19 18:43:23 on dist-test-slave-mld5"
server_key: "f01533bee745dc4e3fb169cde239b38f"
server_key_iv: "f19db04914b7076fa1da356ac90ecf3f"
server_key_version: "encryptionkey@0"
I20260619 18:43:23.865301 32146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:43:23.869557 32146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:43:23.870100 32146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:43:23.870265 32146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:43:23.870749 32146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:43:23.871645 32146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:43:23.871712 32146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:23.871776 32146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:43:23.871829 32146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:23.879810 32146 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.67:43539
I20260619 18:43:23.879832 32273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.67:43539 every 8 connection(s)
I20260619 18:43:23.880576 32146 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
I20260619 18:43:23.886298 31863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:23.882232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:39313 (local address 127.21.143.126:38339)
0619 18:43:23.882390 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:23.882395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:23.883405 (+ 1010us) server_negotiation.cc:407] Connection header received
0619 18:43:23.884646 (+ 1241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:23.884649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:23.884688 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:23.884761 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:23.885779 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:23.885782 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:23.885784 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:43:23.885796 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:23.885815 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:43:23.885823 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:23.885905 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:23.886150 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 18:43:23.886192 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 18:43:23.887099 32276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:23.882624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:38339 (local address 127.21.143.67:39313)
0619 18:43:23.883242 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:23.883279 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:23.884462 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:23.884925 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:23.884932 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:23.885017 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:23.885527 (+ 510us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:43:23.885529 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:23.885567 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:43:23.885633 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:43:23.885976 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:43:23.885984 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:43:23.885989 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 18:43:23.886140 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 18:43:23.886326 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":99,"threads_started":1}
I20260619 18:43:23.888244 32274 heartbeater.cc:347] Connected to a master server at 127.21.143.126:38339
I20260619 18:43:23.888458 32274 heartbeater.cc:464] Registering TS with master...
I20260619 18:43:23.888804 32274 heartbeater.cc:511] Master 127.21.143.126:38339 requested a full tablet report, sending...
I20260619 18:43:23.889405 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 32146
I20260619 18:43:23.889536 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance
I20260619 18:43:23.889817 22077 external_mini_cluster.cc:1428] Setting key da3f1994cd6ff664159b43e7c81399a5
I20260619 18:43:23.889824 31824 ts_manager.cc:194] Registered new tserver with Master: 01bcc5bc6ce54eb19166ba129a28be95 (127.21.143.67:43539)
I20260619 18:43:23.891799 22077 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 18:43:23.894618 31863 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:43:23.893655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33892 (local address 127.21.143.126:38339)
0619 18:43:23.893805 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:23.893811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:23.894020 (+ 209us) server_negotiation.cc:407] Connection header received
0619 18:43:23.894166 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:23.894169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:23.894208 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:23.894292 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:23.894501 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33892: BlockingRecv error: recv got EOF from 127.0.0.1:33892 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 18:43:23.895041 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 31884
I20260619 18:43:23.901762 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 32015
I20260619 18:43:23.906888 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 32146
I20260619 18:43:23.912341 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 31793
2026-06-19T18:43:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (780 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:43:23 dist-test-slave-mld5 krb5kdc[32284](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:43:23 dist-test-slave-mld5 krb5kdc[32284](info): set up 2 sockets
Jun 19 18:43:23 dist-test-slave-mld5 krb5kdc[32284](info): commencing operation
krb5kdc: starting...
W20260619 18:43:24.948351 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:43:25 dist-test-slave-mld5 krb5kdc[32284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:43:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:43:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.143.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
I20260619 18:43:25.061736 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:43829
--webserver_interface=127.21.143.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:39669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:43829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:43:25.167402 32297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:43:25.167719 32297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:43:25.167807 32297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:43:25.171912 32297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:43:25.172010 32297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:43:25.172049 32297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:43:25.172086 32297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:43:25.172115 32297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:43:25.177189 32297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:39669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:43829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:43829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:43:25.178217 32297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:43:25.179251 32297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:43:25.187559 32302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:25.187546 32305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:25.187567 32303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:25.187609 32297 server_base.cc:1034] running on GCE node
I20260619 18:43:25.188402 32297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:43:25.189453 32297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:43:25.190650 32297 hybrid_clock.cc:648] HybridClock initialized: now 1781894605190622 us; error 39 us; skew 500 ppm
Jun 19 18:43:25 dist-test-slave-mld5 krb5kdc[32284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894605, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:43:25.200116 32297 init.cc:377] Logged in from keytab as kudu/127.21.143.126@KRBTEST.COM (short username kudu)
I20260619 18:43:25.201056 32297 webserver.cc:460] Webserver started at http://127.21.143.126:34001/ using document root <none> and password file <none>
I20260619 18:43:25.201618 32297 fs_manager.cc:359] Metadata directory not provided
I20260619 18:43:25.201709 32297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:43:25.201941 32297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:43:25.204047 32297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "9c36be2383e242f1a8f35af9e843c13a"
format_stamp: "Formatted at 2026-06-19 18:43:25 on dist-test-slave-mld5"
server_key: "217deb04a791008d606b1d0d73af0c02"
server_key_iv: "d48c9f24cc88bc450dcef3e5df1cd438"
server_key_version: "encryptionkey@0"
I20260619 18:43:25.204607 32297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "9c36be2383e242f1a8f35af9e843c13a"
format_stamp: "Formatted at 2026-06-19 18:43:25 on dist-test-slave-mld5"
server_key: "217deb04a791008d606b1d0d73af0c02"
server_key_iv: "d48c9f24cc88bc450dcef3e5df1cd438"
server_key_version: "encryptionkey@0"
I20260619 18:43:25.208233 32297 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 18:43:25.210563 32312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:25.211769 32297 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 18:43:25.211908 32297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "9c36be2383e242f1a8f35af9e843c13a"
format_stamp: "Formatted at 2026-06-19 18:43:25 on dist-test-slave-mld5"
server_key: "217deb04a791008d606b1d0d73af0c02"
server_key_iv: "d48c9f24cc88bc450dcef3e5df1cd438"
server_key_version: "encryptionkey@0"
I20260619 18:43:25.212016 32297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:43:25.229725 32297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:43:25.232786 32297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:43:25.232993 32297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:43:25.239725 32297 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:43829
I20260619 18:43:25.240015 32364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:43829 every 8 connection(s)
I20260619 18:43:25.240650 32297 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:43:25.243583 32365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:43:25.247171 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 32297
I20260619 18:43:25.247335 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:43:25.247619 22077 external_mini_cluster.cc:1428] Setting key 0b57c12e8dbb2aa74a41372759852628
I20260619 18:43:25.249146 32365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c36be2383e242f1a8f35af9e843c13a: Bootstrap starting.
I20260619 18:43:25.251607 32365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c36be2383e242f1a8f35af9e843c13a: Neither blocks nor log segments found. Creating new log.
I20260619 18:43:25.252324 32365 log.cc:824] T 00000000000000000000000000000000 P 9c36be2383e242f1a8f35af9e843c13a: Log is configured to *not* fsync() on all Append() calls
I20260619 18:43:25.254289 32365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c36be2383e242f1a8f35af9e843c13a: No bootstrap required, opened a new log
Jun 19 18:43:25 dist-test-slave-mld5 krb5kdc[32284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:43:25.258111 32365 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c36be2383e242f1a8f35af9e843c13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c36be2383e242f1a8f35af9e843c13a" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 43829 } }
I20260619 18:43:25.258340 32365 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c36be2383e242f1a8f35af9e843c13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:43:25.258426 32365 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c36be2383e242f1a8f35af9e843c13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c36be2383e242f1a8f35af9e843c13a, State: Initialized, Role: FOLLOWER
I20260619 18:43:25.258841 32365 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c36be2383e242f1a8f35af9e843c13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c36be2383e242f1a8f35af9e843c13a" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 43829 } }
I20260619 18:43:25.258984 32365 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c36be2383e242f1a8f35af9e843c13a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:43:25.259070 32365 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c36be2383e242f1a8f35af9e843c13a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:43:25.259196 32365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c36be2383e242f1a8f35af9e843c13a [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:43:25.260015 32365 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c36be2383e242f1a8f35af9e843c13a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c36be2383e242f1a8f35af9e843c13a" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 43829 } }
I20260619 18:43:25.260339 32365 leader_election.cc:302] T 00000000000000000000000000000000 P 9c36be2383e242f1a8f35af9e843c13a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c36be2383e242f1a8f35af9e843c13a; no voters:
I20260619 18:43:25.260672 32365 leader_election.cc:288] T 00000000000000000000000000000000 P 9c36be2383e242f1a8f35af9e843c13a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:43:25.260737 32370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c36be2383e242f1a8f35af9e843c13a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:43:25.260936 32370 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c36be2383e242f1a8f35af9e843c13a [term 1 LEADER]: Becoming Leader. State: Replica: 9c36be2383e242f1a8f35af9e843c13a, State: Running, Role: LEADER
I20260619 18:43:25.261220 32370 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c36be2383e242f1a8f35af9e843c13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c36be2383e242f1a8f35af9e843c13a" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 43829 } }
I20260619 18:43:25.261689 32368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:25.248957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57858 (local address 127.21.143.126:43829)
0619 18:43:25.249334 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:25.249353 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:25.249399 (+ 46us) server_negotiation.cc:407] Connection header received
0619 18:43:25.249988 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:25.250003 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:25.250317 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:25.250597 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:25.251507 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:25.252579 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:25.253665 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:25.254113 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:25.254228 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:25.256817 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:25.256822 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:25.256834 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:25.256858 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:25.258708 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:25.259175 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:25.259181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:25.259186 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:25.259362 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:25.259644 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:25.259649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:25.259650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:25.260015 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:25.260221 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:25.260451 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 18:43:25.260755 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":127,"threads_started":1}
I20260619 18:43:25.261989 32365 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c36be2383e242f1a8f35af9e843c13a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:43:25.262403 32370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c36be2383e242f1a8f35af9e843c13a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c36be2383e242f1a8f35af9e843c13a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c36be2383e242f1a8f35af9e843c13a" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 43829 } } }
I20260619 18:43:25.262504 32370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c36be2383e242f1a8f35af9e843c13a [sys.catalog]: This master's current role is: LEADER
I20260619 18:43:25.262871 32374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:43:25.262820 32371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c36be2383e242f1a8f35af9e843c13a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c36be2383e242f1a8f35af9e843c13a. Latest consensus state: current_term: 1 leader_uuid: "9c36be2383e242f1a8f35af9e843c13a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c36be2383e242f1a8f35af9e843c13a" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 43829 } } }
I20260619 18:43:25.262920 32371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c36be2383e242f1a8f35af9e843c13a [sys.catalog]: This master's current role is: LEADER
I20260619 18:43:25.266095 32374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:43:25.271531 32374 catalog_manager.cc:1347] Generated new cluster ID: d1496a0863ff4b888c946dbc044e0235
I20260619 18:43:25.271593 32374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:43:25.289143 32374 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:43:25.290057 32374 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:43:25.301101 32374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c36be2383e242f1a8f35af9e843c13a: Generated new TSK 0
I20260619 18:43:25.301782 32374 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.143.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab.
Entry for principal kudu/127.21.143.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab.
I20260619 18:43:25.339123 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.65:0
--local_ip_for_outbound_sockets=127.21.143.65
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:43829
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:39669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:43:25.451428 32391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:43:25.451675 32391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:43:25.451752 32391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:43:25.455885 32391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:43:25.455940 32391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:43:25.456048 32391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.65
I20260619 18:43:25.461254 32391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:39669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:43829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.65
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:43:25.462349 32391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:43:25.463502 32391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:43:25.472872 32396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:25.472911 32397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:25.472906 32399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:25.473121 32391 server_base.cc:1034] running on GCE node
I20260619 18:43:25.473702 32391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:43:25.474319 32391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:43:25.475483 32391 hybrid_clock.cc:648] HybridClock initialized: now 1781894605475462 us; error 34 us; skew 500 ppm
Jun 19 18:43:25 dist-test-slave-mld5 krb5kdc[32284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894605, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:43:25.478152 32391 init.cc:377] Logged in from keytab as kudu/127.21.143.65@KRBTEST.COM (short username kudu)
I20260619 18:43:25.478947 32391 webserver.cc:460] Webserver started at http://127.21.143.65:33965/ using document root <none> and password file <none>
I20260619 18:43:25.479532 32391 fs_manager.cc:359] Metadata directory not provided
I20260619 18:43:25.479593 32391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:43:25.479764 32391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:43:25.481729 32391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/data/instance:
uuid: "938d1fb62bf04e448806046383b864c0"
format_stamp: "Formatted at 2026-06-19 18:43:25 on dist-test-slave-mld5"
server_key: "c12539511e4ed644c8dbc554470bef97"
server_key_iv: "2163ba5c410da740300f86ebb20eb173"
server_key_version: "encryptionkey@0"
I20260619 18:43:25.482208 32391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance:
uuid: "938d1fb62bf04e448806046383b864c0"
format_stamp: "Formatted at 2026-06-19 18:43:25 on dist-test-slave-mld5"
server_key: "c12539511e4ed644c8dbc554470bef97"
server_key_iv: "2163ba5c410da740300f86ebb20eb173"
server_key_version: "encryptionkey@0"
I20260619 18:43:25.485780 32391 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:43:25.488065 32406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:25.489229 32391 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:43:25.489326 32391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/wal
uuid: "938d1fb62bf04e448806046383b864c0"
format_stamp: "Formatted at 2026-06-19 18:43:25 on dist-test-slave-mld5"
server_key: "c12539511e4ed644c8dbc554470bef97"
server_key_iv: "2163ba5c410da740300f86ebb20eb173"
server_key_version: "encryptionkey@0"
I20260619 18:43:25.489406 32391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:43:25.503276 32391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:43:25.506431 32391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:43:25.506574 32391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:43:25.507076 32391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:43:25.507882 32391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:43:25.507926 32391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:25.507964 32391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:43:25.507984 32391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:25.516693 32391 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.65:34801
I20260619 18:43:25.516765 32519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.65:34801 every 8 connection(s)
I20260619 18:43:25.517592 32391 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
I20260619 18:43:25.525233 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 32391
I20260619 18:43:25.525362 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance
I20260619 18:43:25.525648 22077 external_mini_cluster.cc:1428] Setting key eb0f137b3464fc6ee2f1ef7e6d21c5bd
Jun 19 18:43:25 dist-test-slave-mld5 krb5kdc[32284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894605, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:43:25.531674 32368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:25.519397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:51165 (local address 127.21.143.126:43829)
0619 18:43:25.519532 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:25.519535 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:25.520267 (+ 732us) server_negotiation.cc:407] Connection header received
0619 18:43:25.521646 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:25.521650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:25.521706 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:25.521808 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:25.522905 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:25.523495 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:25.524630 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:25.524861 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:25.524979 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:25.527820 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:25.527827 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:25.527833 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:25.527866 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:25.529347 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:25.529930 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:25.529932 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:25.529934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:25.530045 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:25.530394 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:25.530401 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:25.530405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:25.530671 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:25.530773 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:25.531273 (+ 500us) server_negotiation.cc:299] Negotiation successful
0619 18:43:25.531433 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 18:43:25.532469 32522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:25.519736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:43829 (local address 127.21.143.65:51165)
0619 18:43:25.520135 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:25.520172 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:25.521437 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:25.521976 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:25.521983 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:25.522313 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:25.522764 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:25.522777 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:25.523620 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:25.523628 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:25.524503 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:25.524511 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:25.525016 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:25.525021 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:25.525190 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:25.525866 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:25.525888 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:25.527655 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:43:25.529457 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:25.529465 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:25.529481 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:25.529829 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:25.530172 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:25.530174 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:25.530175 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:25.530300 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:25.530791 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:43:25.530795 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:43:25.531037 (+ 242us) client_negotiation.cc:769] Sending connection context
0619 18:43:25.531190 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 18:43:25.531592 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":99,"threads_started":1}
I20260619 18:43:25.533660 32520 heartbeater.cc:347] Connected to a master server at 127.21.143.126:43829
I20260619 18:43:25.533885 32520 heartbeater.cc:464] Registering TS with master...
I20260619 18:43:25.534339 32520 heartbeater.cc:511] Master 127.21.143.126:43829 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.143.66@KRBTEST.COM; defaulting to no policy
I20260619 18:43:25.535959 32329 ts_manager.cc:194] Registered new tserver with Master: 938d1fb62bf04e448806046383b864c0 (127.21.143.65:34801)
I20260619 18:43:25.537586 32329 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.65@KRBTEST.COM'} at 127.21.143.65:51165
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab.
Entry for principal kudu/127.21.143.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab.
I20260619 18:43:25.552035 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.66:0
--local_ip_for_outbound_sockets=127.21.143.66
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:43829
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:39669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:43:25.659548 32525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:43:25.659795 32525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:43:25.659883 32525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:43:25.664039 32525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:43:25.664098 32525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:43:25.664204 32525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.66
I20260619 18:43:25.669518 32525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:39669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:43829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.66
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:43:25.670555 32525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:43:25.671595 32525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:43:25.680713 32531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:25.680727 32530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:25.680811 32525 server_base.cc:1034] running on GCE node
W20260619 18:43:25.680980 32533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:25.681434 32525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:43:25.682037 32525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:43:25.683233 32525 hybrid_clock.cc:648] HybridClock initialized: now 1781894605683212 us; error 35 us; skew 500 ppm
Jun 19 18:43:25 dist-test-slave-mld5 krb5kdc[32284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894605, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:43:25.685818 32525 init.cc:377] Logged in from keytab as kudu/127.21.143.66@KRBTEST.COM (short username kudu)
I20260619 18:43:25.686546 32525 webserver.cc:460] Webserver started at http://127.21.143.66:36147/ using document root <none> and password file <none>
I20260619 18:43:25.687052 32525 fs_manager.cc:359] Metadata directory not provided
I20260619 18:43:25.687103 32525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:43:25.687255 32525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:43:25.689167 32525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/data/instance:
uuid: "863a0e5a8a294a52b82f829980953745"
format_stamp: "Formatted at 2026-06-19 18:43:25 on dist-test-slave-mld5"
server_key: "a26dc498b490a2116d4a99d37e8ae87a"
server_key_iv: "819895fff41ce3c5196fb768ba61c573"
server_key_version: "encryptionkey@0"
I20260619 18:43:25.689651 32525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance:
uuid: "863a0e5a8a294a52b82f829980953745"
format_stamp: "Formatted at 2026-06-19 18:43:25 on dist-test-slave-mld5"
server_key: "a26dc498b490a2116d4a99d37e8ae87a"
server_key_iv: "819895fff41ce3c5196fb768ba61c573"
server_key_version: "encryptionkey@0"
I20260619 18:43:25.693286 32525 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:43:25.695588 32540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:25.696741 32525 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:43:25.696832 32525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/wal
uuid: "863a0e5a8a294a52b82f829980953745"
format_stamp: "Formatted at 2026-06-19 18:43:25 on dist-test-slave-mld5"
server_key: "a26dc498b490a2116d4a99d37e8ae87a"
server_key_iv: "819895fff41ce3c5196fb768ba61c573"
server_key_version: "encryptionkey@0"
I20260619 18:43:25.696954 32525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:43:25.705844 32525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:43:25.708707 32525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:43:25.708890 32525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:43:25.709446 32525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:43:25.710216 32525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:43:25.710261 32525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:25.710323 32525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:43:25.710357 32525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:25.718904 32525 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.66:40213
I20260619 18:43:25.718930 32653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.66:40213 every 8 connection(s)
I20260619 18:43:25.719762 32525 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
I20260619 18:43:25.727453 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 32525
I20260619 18:43:25.727572 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance
I20260619 18:43:25.727849 22077 external_mini_cluster.cc:1428] Setting key 8847eeb29eba883b4760b3f954a0c250
Jun 19 18:43:25 dist-test-slave-mld5 krb5kdc[32284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894605, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:43:25.732895 32368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:25.721521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:33083 (local address 127.21.143.126:43829)
0619 18:43:25.721690 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:25.721694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:25.722528 (+ 834us) server_negotiation.cc:407] Connection header received
0619 18:43:25.723442 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:25.723445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:25.723504 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:25.723610 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:25.724733 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:25.725269 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:25.726363 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:25.726587 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:25.726715 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:25.729353 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:25.729357 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:25.729360 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:25.729391 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:25.730796 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:25.731350 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:25.731353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:25.731355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:25.731435 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:25.731773 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:25.731778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:25.731780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:25.731956 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:25.732074 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:25.732619 (+ 545us) server_negotiation.cc:299] Negotiation successful
0619 18:43:25.732733 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 18:43:25.733716 32656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:25.721767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:43829 (local address 127.21.143.66:33083)
0619 18:43:25.722381 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:25.722419 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:25.723207 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:25.723774 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:25.723781 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:25.724140 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:25.724589 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:25.724602 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:25.725405 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:25.725408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:25.726240 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:25.726249 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:25.726748 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:25.726752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:25.726920 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:25.727557 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:25.727584 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:25.729187 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:43:25.730924 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:25.730930 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:25.730943 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:25.731222 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:25.731531 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:25.731542 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:25.731545 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:25.731679 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:25.732072 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:43:25.732079 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:43:25.732347 (+ 268us) client_negotiation.cc:769] Sending connection context
0619 18:43:25.732574 (+ 227us) client_negotiation.cc:240] Negotiation successful
0619 18:43:25.732864 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":100,"threads_started":1}
I20260619 18:43:25.734805 32654 heartbeater.cc:347] Connected to a master server at 127.21.143.126:43829
I20260619 18:43:25.735055 32654 heartbeater.cc:464] Registering TS with master...
I20260619 18:43:25.735603 32654 heartbeater.cc:511] Master 127.21.143.126:43829 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.143.67@KRBTEST.COM; defaulting to no policy
I20260619 18:43:25.736677 32329 ts_manager.cc:194] Registered new tserver with Master: 863a0e5a8a294a52b82f829980953745 (127.21.143.66:40213)
I20260619 18:43:25.737233 32329 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.66@KRBTEST.COM'} at 127.21.143.66:33083
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab.
Entry for principal kudu/127.21.143.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab.
I20260619 18:43:25.754083 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.67:0
--local_ip_for_outbound_sockets=127.21.143.67
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:43829
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:39669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:43:25.862716 32659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:43:25.863034 32659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:43:25.863082 32659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:43:25.867276 32659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:43:25.867353 32659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:43:25.867414 32659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.67
I20260619 18:43:25.872859 32659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:39669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:43829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.67
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:43:25.873936 32659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:43:25.874961 32659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:43:25.884744 32667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:25.884768 32664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:25.884733 32665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:25.885106 32659 server_base.cc:1034] running on GCE node
I20260619 18:43:25.885519 32659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:43:25.886085 32659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:43:25.887241 32659 hybrid_clock.cc:648] HybridClock initialized: now 1781894605887224 us; error 26 us; skew 500 ppm
Jun 19 18:43:25 dist-test-slave-mld5 krb5kdc[32284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894605, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:43:25.889883 32659 init.cc:377] Logged in from keytab as kudu/127.21.143.67@KRBTEST.COM (short username kudu)
I20260619 18:43:25.890753 32659 webserver.cc:460] Webserver started at http://127.21.143.67:45641/ using document root <none> and password file <none>
I20260619 18:43:25.891369 32659 fs_manager.cc:359] Metadata directory not provided
I20260619 18:43:25.891456 32659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:43:25.891690 32659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:43:25.893777 32659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/data/instance:
uuid: "83902cee8ba9431989cb5e9a7e9f2696"
format_stamp: "Formatted at 2026-06-19 18:43:25 on dist-test-slave-mld5"
server_key: "e3799743fe8574fe40b4641fcfeecbfd"
server_key_iv: "ff8a9e5c3daf0534d11e16485984eff9"
server_key_version: "encryptionkey@0"
I20260619 18:43:25.894345 32659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance:
uuid: "83902cee8ba9431989cb5e9a7e9f2696"
format_stamp: "Formatted at 2026-06-19 18:43:25 on dist-test-slave-mld5"
server_key: "e3799743fe8574fe40b4641fcfeecbfd"
server_key_iv: "ff8a9e5c3daf0534d11e16485984eff9"
server_key_version: "encryptionkey@0"
I20260619 18:43:25.897890 32659 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 18:43:25.900218 32674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:25.901525 32659 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:43:25.901660 32659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/wal
uuid: "83902cee8ba9431989cb5e9a7e9f2696"
format_stamp: "Formatted at 2026-06-19 18:43:25 on dist-test-slave-mld5"
server_key: "e3799743fe8574fe40b4641fcfeecbfd"
server_key_iv: "ff8a9e5c3daf0534d11e16485984eff9"
server_key_version: "encryptionkey@0"
I20260619 18:43:25.901773 32659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:43:25.912665 32659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:43:25.915280 32659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:43:25.915534 32659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:43:25.916100 32659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:43:25.916909 32659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:43:25.916985 32659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:25.917061 32659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:43:25.917115 32659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:25.926019 32659 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.67:44159
I20260619 18:43:25.926043 319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.67:44159 every 8 connection(s)
I20260619 18:43:25.926853 32659 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
I20260619 18:43:25.930392 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 32659
I20260619 18:43:25.930510 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance
I20260619 18:43:25.930812 22077 external_mini_cluster.cc:1428] Setting key c953bd69d4af5ed46a9e4e35e5c4e1d7
Jun 19 18:43:25 dist-test-slave-mld5 krb5kdc[32284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894605, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:43:25.939906 32368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:25.928532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:41185 (local address 127.21.143.126:43829)
0619 18:43:25.928668 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:25.928671 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:25.929392 (+ 721us) server_negotiation.cc:407] Connection header received
0619 18:43:25.931033 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:25.931036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:25.931084 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:25.931202 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:25.932636 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:25.933176 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:25.934387 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:25.934613 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:25.934725 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:25.937071 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:25.937074 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:25.937077 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:25.937099 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:25.938256 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:25.938680 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:25.938682 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:25.938684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:25.938751 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:25.939013 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:25.939015 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:25.939016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:25.939166 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:25.939235 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:25.939703 (+ 468us) server_negotiation.cc:299] Negotiation successful
0619 18:43:25.939791 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 18:43:25.940680 322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:25.928833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:43829 (local address 127.21.143.67:41185)
0619 18:43:25.929251 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:25.929290 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:25.930799 (+ 1509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:25.931413 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:25.931422 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:25.931923 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:25.932495 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:25.932509 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:25.933296 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:25.933300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:25.934255 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:25.934264 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:25.934760 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:25.934765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:25.934936 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:25.935545 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:25.935561 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:25.936935 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:43:25.938348 (+ 1413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:25.938352 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:25.938363 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:25.938595 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:25.938822 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:25.938824 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:25.938825 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:25.938942 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:25.939245 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:43:25.939249 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:43:25.939503 (+ 254us) client_negotiation.cc:769] Sending connection context
0619 18:43:25.939662 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 18:43:25.939886 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":105,"threads_started":1}
I20260619 18:43:25.941774 320 heartbeater.cc:347] Connected to a master server at 127.21.143.126:43829
I20260619 18:43:25.941996 320 heartbeater.cc:464] Registering TS with master...
I20260619 18:43:25.942448 320 heartbeater.cc:511] Master 127.21.143.126:43829 requested a full tablet report, sending...
I20260619 18:43:25.943424 32329 ts_manager.cc:194] Registered new tserver with Master: 83902cee8ba9431989cb5e9a7e9f2696 (127.21.143.67:44159)
I20260619 18:43:25.944015 32329 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.67@KRBTEST.COM'} at 127.21.143.67:41185
I20260619 18:43:25.945067 22077 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:43:25.953114 32368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:25.947041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57864 (local address 127.21.143.126:43829)
0619 18:43:25.947145 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:25.947149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:25.947344 (+ 195us) server_negotiation.cc:407] Connection header received
0619 18:43:25.947448 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:25.947450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:25.947483 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:25.947544 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:25.948124 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:25.948582 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:25.949261 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:25.949436 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:25.949499 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:25.950654 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:25.950656 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:25.950658 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:25.950675 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:25.951857 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:25.952208 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:25.952210 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:25.952212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:25.952276 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:25.952497 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:25.952499 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:25.952500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:25.952679 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:25.952771 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:25.952935 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 18:43:25.953013 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 18:43:25.955768 32329 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:43:25.958640 32329 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:43:25.972426 331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:25.966665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:40213 (local address 127.0.0.1:44852)
0619 18:43:25.967056 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:25.967109 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:25.967258 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:25.968044 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:25.968051 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:25.968075 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:43:25.968212 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:25.968221 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:25.969653 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:25.969657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:25.970985 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:25.970995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:25.971888 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:25.971891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:25.971989 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:25.972028 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 18:43:25.972138 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 18:43:25.972280 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":72,"threads_started":1}
I20260619 18:43:25.972426 332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:25.966777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44852 (local address 127.21.143.66:40213)
0619 18:43:25.967067 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:25.967080 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:25.967221 (+ 141us) server_negotiation.cc:407] Connection header received
0619 18:43:25.967442 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:25.967448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:25.967670 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:25.967839 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:43:25.968346 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:25.969510 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:25.971129 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:25.971766 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:25.971887 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:25.972033 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:25.972177 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 18:43:25.972269 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":100,"threads_started":1}
I20260619 18:43:25.972623 333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:25.967576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:34801 (local address 127.0.0.1:48650)
0619 18:43:25.967837 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:25.967851 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:25.967939 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:25.968818 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:25.968822 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:25.968837 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:43:25.968934 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:25.968943 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:25.970099 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:25.970103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:25.971168 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:25.971180 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:25.972284 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:25.972287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:25.972369 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:25.972388 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 18:43:25.972435 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 18:43:25.972504 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":112,"threads_started":1}
I20260619 18:43:25.973202 334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:25.967599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48650 (local address 127.21.143.65:34801)
0619 18:43:25.968369 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:25.968387 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:25.968410 (+ 23us) server_negotiation.cc:407] Connection header received
0619 18:43:25.968481 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:25.968486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:25.968681 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:25.968828 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:43:25.969053 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:25.969967 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:25.971293 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:25.972160 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:25.972787 (+ 627us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:25.972931 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:25.972978 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 18:43:25.973065 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":130,"threads_started":1}
I20260619 18:43:25.974005 335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:25.967895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:44159 (local address 127.0.0.1:39152)
0619 18:43:25.968123 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:25.968134 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:25.968219 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:25.969015 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:25.969018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:25.969035 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:43:25.969112 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:25.969117 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:25.970643 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:25.970646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:25.971736 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:25.971746 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:25.973624 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:25.973628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:25.973713 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:25.973735 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 18:43:25.973808 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 18:43:25.973875 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":78,"threads_started":1}
I20260619 18:43:25.974642 336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:25.968268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39152 (local address 127.21.143.67:44159)
0619 18:43:25.968590 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:25.968604 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:25.968631 (+ 27us) server_negotiation.cc:407] Connection header received
0619 18:43:25.968689 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:25.968693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:25.968865 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:25.969224 (+ 359us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:43:25.969329 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:25.970312 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:25.971872 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:25.973499 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:25.973607 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:25.973726 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:25.974403 (+ 677us) server_negotiation.cc:299] Negotiation successful
0619 18:43:25.974503 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":145,"threads_started":1}
I20260619 18:43:25.975562 32588 tablet_service.cc:1467] Processing CreateTablet for tablet b242bc596f1c4c03a6972ad0b32614b3 (DEFAULT_TABLE table=test-table [id=afaef5fdc4e74a94a056e8a5a7630c9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:43:25.975911 32454 tablet_service.cc:1467] Processing CreateTablet for tablet b242bc596f1c4c03a6972ad0b32614b3 (DEFAULT_TABLE table=test-table [id=afaef5fdc4e74a94a056e8a5a7630c9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:43:25.976430 32588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b242bc596f1c4c03a6972ad0b32614b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:43:25.976854 32454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b242bc596f1c4c03a6972ad0b32614b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:43:25.977066 32722 tablet_service.cc:1467] Processing CreateTablet for tablet b242bc596f1c4c03a6972ad0b32614b3 (DEFAULT_TABLE table=test-table [id=afaef5fdc4e74a94a056e8a5a7630c9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:43:25.978004 32722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b242bc596f1c4c03a6972ad0b32614b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:43:25.983193 337 tablet_bootstrap.cc:492] T b242bc596f1c4c03a6972ad0b32614b3 P 863a0e5a8a294a52b82f829980953745: Bootstrap starting.
I20260619 18:43:25.984238 338 tablet_bootstrap.cc:492] T b242bc596f1c4c03a6972ad0b32614b3 P 938d1fb62bf04e448806046383b864c0: Bootstrap starting.
I20260619 18:43:25.985694 337 tablet_bootstrap.cc:654] T b242bc596f1c4c03a6972ad0b32614b3 P 863a0e5a8a294a52b82f829980953745: Neither blocks nor log segments found. Creating new log.
I20260619 18:43:25.986402 337 log.cc:824] T b242bc596f1c4c03a6972ad0b32614b3 P 863a0e5a8a294a52b82f829980953745: Log is configured to *not* fsync() on all Append() calls
I20260619 18:43:25.986526 338 tablet_bootstrap.cc:654] T b242bc596f1c4c03a6972ad0b32614b3 P 938d1fb62bf04e448806046383b864c0: Neither blocks nor log segments found. Creating new log.
I20260619 18:43:25.987175 338 log.cc:824] T b242bc596f1c4c03a6972ad0b32614b3 P 938d1fb62bf04e448806046383b864c0: Log is configured to *not* fsync() on all Append() calls
I20260619 18:43:25.987622 339 tablet_bootstrap.cc:492] T b242bc596f1c4c03a6972ad0b32614b3 P 83902cee8ba9431989cb5e9a7e9f2696: Bootstrap starting.
I20260619 18:43:25.988416 337 tablet_bootstrap.cc:492] T b242bc596f1c4c03a6972ad0b32614b3 P 863a0e5a8a294a52b82f829980953745: No bootstrap required, opened a new log
I20260619 18:43:25.988607 337 ts_tablet_manager.cc:1397] T b242bc596f1c4c03a6972ad0b32614b3 P 863a0e5a8a294a52b82f829980953745: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 18:43:25.989043 338 tablet_bootstrap.cc:492] T b242bc596f1c4c03a6972ad0b32614b3 P 938d1fb62bf04e448806046383b864c0: No bootstrap required, opened a new log
I20260619 18:43:25.989210 338 ts_tablet_manager.cc:1397] T b242bc596f1c4c03a6972ad0b32614b3 P 938d1fb62bf04e448806046383b864c0: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260619 18:43:25.989638 339 tablet_bootstrap.cc:654] T b242bc596f1c4c03a6972ad0b32614b3 P 83902cee8ba9431989cb5e9a7e9f2696: Neither blocks nor log segments found. Creating new log.
I20260619 18:43:25.990559 339 log.cc:824] T b242bc596f1c4c03a6972ad0b32614b3 P 83902cee8ba9431989cb5e9a7e9f2696: Log is configured to *not* fsync() on all Append() calls
I20260619 18:43:25.992151 339 tablet_bootstrap.cc:492] T b242bc596f1c4c03a6972ad0b32614b3 P 83902cee8ba9431989cb5e9a7e9f2696: No bootstrap required, opened a new log
I20260619 18:43:25.992338 339 ts_tablet_manager.cc:1397] T b242bc596f1c4c03a6972ad0b32614b3 P 83902cee8ba9431989cb5e9a7e9f2696: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 18:43:25.992483 337 raft_consensus.cc:348] T b242bc596f1c4c03a6972ad0b32614b3 P 863a0e5a8a294a52b82f829980953745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "863a0e5a8a294a52b82f829980953745" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 40213 } } peers { permanent_uuid: "938d1fb62bf04e448806046383b864c0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 34801 } } peers { permanent_uuid: "83902cee8ba9431989cb5e9a7e9f2696" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 44159 } }
I20260619 18:43:25.992802 337 raft_consensus.cc:374] T b242bc596f1c4c03a6972ad0b32614b3 P 863a0e5a8a294a52b82f829980953745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:43:25.992861 337 raft_consensus.cc:729] T b242bc596f1c4c03a6972ad0b32614b3 P 863a0e5a8a294a52b82f829980953745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 863a0e5a8a294a52b82f829980953745, State: Initialized, Role: FOLLOWER
I20260619 18:43:25.992944 338 raft_consensus.cc:348] T b242bc596f1c4c03a6972ad0b32614b3 P 938d1fb62bf04e448806046383b864c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "863a0e5a8a294a52b82f829980953745" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 40213 } } peers { permanent_uuid: "938d1fb62bf04e448806046383b864c0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 34801 } } peers { permanent_uuid: "83902cee8ba9431989cb5e9a7e9f2696" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 44159 } }
I20260619 18:43:25.993191 338 raft_consensus.cc:374] T b242bc596f1c4c03a6972ad0b32614b3 P 938d1fb62bf04e448806046383b864c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:43:25.993252 338 raft_consensus.cc:729] T b242bc596f1c4c03a6972ad0b32614b3 P 938d1fb62bf04e448806046383b864c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 938d1fb62bf04e448806046383b864c0, State: Initialized, Role: FOLLOWER
I20260619 18:43:25.993355 337 consensus_queue.cc:260] T b242bc596f1c4c03a6972ad0b32614b3 P 863a0e5a8a294a52b82f829980953745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "863a0e5a8a294a52b82f829980953745" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 40213 } } peers { permanent_uuid: "938d1fb62bf04e448806046383b864c0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 34801 } } peers { permanent_uuid: "83902cee8ba9431989cb5e9a7e9f2696" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 44159 } }
I20260619 18:43:25.993677 338 consensus_queue.cc:260] T b242bc596f1c4c03a6972ad0b32614b3 P 938d1fb62bf04e448806046383b864c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "863a0e5a8a294a52b82f829980953745" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 40213 } } peers { permanent_uuid: "938d1fb62bf04e448806046383b864c0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 34801 } } peers { permanent_uuid: "83902cee8ba9431989cb5e9a7e9f2696" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 44159 } }
I20260619 18:43:25.994065 32654 heartbeater.cc:503] Master 127.21.143.126:43829 was elected leader, sending a full tablet report...
I20260619 18:43:25.994375 337 ts_tablet_manager.cc:1428] T b242bc596f1c4c03a6972ad0b32614b3 P 863a0e5a8a294a52b82f829980953745: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 18:43:25.994436 338 ts_tablet_manager.cc:1428] T b242bc596f1c4c03a6972ad0b32614b3 P 938d1fb62bf04e448806046383b864c0: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260619 18:43:25.994724 32520 heartbeater.cc:503] Master 127.21.143.126:43829 was elected leader, sending a full tablet report...
I20260619 18:43:25.996304 339 raft_consensus.cc:348] T b242bc596f1c4c03a6972ad0b32614b3 P 83902cee8ba9431989cb5e9a7e9f2696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "863a0e5a8a294a52b82f829980953745" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 40213 } } peers { permanent_uuid: "938d1fb62bf04e448806046383b864c0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 34801 } } peers { permanent_uuid: "83902cee8ba9431989cb5e9a7e9f2696" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 44159 } }
I20260619 18:43:25.996539 339 raft_consensus.cc:374] T b242bc596f1c4c03a6972ad0b32614b3 P 83902cee8ba9431989cb5e9a7e9f2696 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:43:25.996595 339 raft_consensus.cc:729] T b242bc596f1c4c03a6972ad0b32614b3 P 83902cee8ba9431989cb5e9a7e9f2696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83902cee8ba9431989cb5e9a7e9f2696, State: Initialized, Role: FOLLOWER
I20260619 18:43:25.997007 339 consensus_queue.cc:260] T b242bc596f1c4c03a6972ad0b32614b3 P 83902cee8ba9431989cb5e9a7e9f2696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "863a0e5a8a294a52b82f829980953745" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 40213 } } peers { permanent_uuid: "938d1fb62bf04e448806046383b864c0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 34801 } } peers { permanent_uuid: "83902cee8ba9431989cb5e9a7e9f2696" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 44159 } }
I20260619 18:43:25.997555 320 heartbeater.cc:503] Master 127.21.143.126:43829 was elected leader, sending a full tablet report...
I20260619 18:43:25.997908 339 ts_tablet_manager.cc:1428] T b242bc596f1c4c03a6972ad0b32614b3 P 83902cee8ba9431989cb5e9a7e9f2696: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
W20260619 18:43:26.018754 32521 tablet.cc:2369] T b242bc596f1c4c03a6972ad0b32614b3 P 938d1fb62bf04e448806046383b864c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:43:26.143796 345 raft_consensus.cc:482] T b242bc596f1c4c03a6972ad0b32614b3 P 83902cee8ba9431989cb5e9a7e9f2696 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:43:26.144037 345 raft_consensus.cc:504] T b242bc596f1c4c03a6972ad0b32614b3 P 83902cee8ba9431989cb5e9a7e9f2696 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "863a0e5a8a294a52b82f829980953745" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 40213 } } peers { permanent_uuid: "938d1fb62bf04e448806046383b864c0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 34801 } } peers { permanent_uuid: "83902cee8ba9431989cb5e9a7e9f2696" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 44159 } }
I20260619 18:43:26.145349 345 leader_election.cc:288] T b242bc596f1c4c03a6972ad0b32614b3 P 83902cee8ba9431989cb5e9a7e9f2696 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 863a0e5a8a294a52b82f829980953745 (127.21.143.66:40213), 938d1fb62bf04e448806046383b864c0 (127.21.143.65:34801)
I20260619 18:43:26.149824 322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:26.145627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:40213 (local address 127.21.143.67:53589)
0619 18:43:26.145781 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:26.145798 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:26.145900 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:26.146233 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:26.146236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:26.146256 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:43:26.146383 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:26.146392 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:26.147434 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:26.147437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:26.148593 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:26.148602 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:26.149407 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:26.149409 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:26.149512 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:26.149536 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 18:43:26.149610 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 18:43:26.149688 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":1}
I20260619 18:43:26.149863 346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:26.145968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:34801 (local address 127.21.143.67:52381)
0619 18:43:26.146332 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:26.146348 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:26.146442 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:26.146695 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:26.146696 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:26.146709 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:43:26.146815 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:26.146820 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:26.147713 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:26.147715 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:26.148777 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:26.148783 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:26.149521 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:26.149524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:26.149631 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:26.149648 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 18:43:26.149710 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 18:43:26.149761 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":1,"thread_start_us":112,"threads_started":1}
I20260619 18:43:26.149853 332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:26.145759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:53589 (local address 127.21.143.66:40213)
0619 18:43:26.145905 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:26.145909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:26.145946 (+ 37us) server_negotiation.cc:407] Connection header received
0619 18:43:26.146007 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:26.146010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:26.146097 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:26.146202 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:43:26.146518 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:26.147344 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:26.148716 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:26.149302 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:26.149424 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:26.149494 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:26.149643 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 18:43:26.149735 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 18:43:26.149909 334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:26.146104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:52381 (local address 127.21.143.65:34801)
0619 18:43:26.146242 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:26.146247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:26.146424 (+ 177us) server_negotiation.cc:407] Connection header received
0619 18:43:26.146543 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:26.146552 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:26.146600 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:26.146678 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:43:26.146951 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:26.147614 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:26.148872 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:26.149421 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:26.149525 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:26.149574 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:26.149741 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 18:43:26.149803 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 18:43:26.150450 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b242bc596f1c4c03a6972ad0b32614b3" candidate_uuid: "83902cee8ba9431989cb5e9a7e9f2696" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "863a0e5a8a294a52b82f829980953745" is_pre_election: true
I20260619 18:43:26.150749 32608 raft_consensus.cc:2454] T b242bc596f1c4c03a6972ad0b32614b3 P 863a0e5a8a294a52b82f829980953745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83902cee8ba9431989cb5e9a7e9f2696 in term 0.
I20260619 18:43:26.150754 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b242bc596f1c4c03a6972ad0b32614b3" candidate_uuid: "83902cee8ba9431989cb5e9a7e9f2696" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "938d1fb62bf04e448806046383b864c0" is_pre_election: true
I20260619 18:43:26.151051 32474 raft_consensus.cc:2454] T b242bc596f1c4c03a6972ad0b32614b3 P 938d1fb62bf04e448806046383b864c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83902cee8ba9431989cb5e9a7e9f2696 in term 0.
I20260619 18:43:26.151232 32676 leader_election.cc:302] T b242bc596f1c4c03a6972ad0b32614b3 P 83902cee8ba9431989cb5e9a7e9f2696 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83902cee8ba9431989cb5e9a7e9f2696, 863a0e5a8a294a52b82f829980953745; no voters:
I20260619 18:43:26.151566 345 raft_consensus.cc:2781] T b242bc596f1c4c03a6972ad0b32614b3 P 83902cee8ba9431989cb5e9a7e9f2696 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:43:26.151664 345 raft_consensus.cc:482] T b242bc596f1c4c03a6972ad0b32614b3 P 83902cee8ba9431989cb5e9a7e9f2696 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:43:26.151872 345 raft_consensus.cc:3037] T b242bc596f1c4c03a6972ad0b32614b3 P 83902cee8ba9431989cb5e9a7e9f2696 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:43:26.153080 345 raft_consensus.cc:504] T b242bc596f1c4c03a6972ad0b32614b3 P 83902cee8ba9431989cb5e9a7e9f2696 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "863a0e5a8a294a52b82f829980953745" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 40213 } } peers { permanent_uuid: "938d1fb62bf04e448806046383b864c0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 34801 } } peers { permanent_uuid: "83902cee8ba9431989cb5e9a7e9f2696" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 44159 } }
I20260619 18:43:26.153406 345 leader_election.cc:288] T b242bc596f1c4c03a6972ad0b32614b3 P 83902cee8ba9431989cb5e9a7e9f2696 [CANDIDATE]: Term 1 election: Requested vote from peers 863a0e5a8a294a52b82f829980953745 (127.21.143.66:40213), 938d1fb62bf04e448806046383b864c0 (127.21.143.65:34801)
I20260619 18:43:26.153724 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b242bc596f1c4c03a6972ad0b32614b3" candidate_uuid: "83902cee8ba9431989cb5e9a7e9f2696" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "863a0e5a8a294a52b82f829980953745"
I20260619 18:43:26.153796 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b242bc596f1c4c03a6972ad0b32614b3" candidate_uuid: "83902cee8ba9431989cb5e9a7e9f2696" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "938d1fb62bf04e448806046383b864c0"
I20260619 18:43:26.153843 32608 raft_consensus.cc:3037] T b242bc596f1c4c03a6972ad0b32614b3 P 863a0e5a8a294a52b82f829980953745 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:43:26.153910 32474 raft_consensus.cc:3037] T b242bc596f1c4c03a6972ad0b32614b3 P 938d1fb62bf04e448806046383b864c0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:43:26.155109 32474 raft_consensus.cc:2454] T b242bc596f1c4c03a6972ad0b32614b3 P 938d1fb62bf04e448806046383b864c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83902cee8ba9431989cb5e9a7e9f2696 in term 1.
I20260619 18:43:26.155109 32608 raft_consensus.cc:2454] T b242bc596f1c4c03a6972ad0b32614b3 P 863a0e5a8a294a52b82f829980953745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83902cee8ba9431989cb5e9a7e9f2696 in term 1.
I20260619 18:43:26.155380 32676 leader_election.cc:302] T b242bc596f1c4c03a6972ad0b32614b3 P 83902cee8ba9431989cb5e9a7e9f2696 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83902cee8ba9431989cb5e9a7e9f2696, 863a0e5a8a294a52b82f829980953745; no voters:
I20260619 18:43:26.155584 345 raft_consensus.cc:2781] T b242bc596f1c4c03a6972ad0b32614b3 P 83902cee8ba9431989cb5e9a7e9f2696 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:43:26.155807 345 raft_consensus.cc:686] T b242bc596f1c4c03a6972ad0b32614b3 P 83902cee8ba9431989cb5e9a7e9f2696 [term 1 LEADER]: Becoming Leader. State: Replica: 83902cee8ba9431989cb5e9a7e9f2696, State: Running, Role: LEADER
I20260619 18:43:26.156105 345 consensus_queue.cc:237] T b242bc596f1c4c03a6972ad0b32614b3 P 83902cee8ba9431989cb5e9a7e9f2696 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "863a0e5a8a294a52b82f829980953745" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 40213 } } peers { permanent_uuid: "938d1fb62bf04e448806046383b864c0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 34801 } } peers { permanent_uuid: "83902cee8ba9431989cb5e9a7e9f2696" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 44159 } }
I20260619 18:43:26.159117 32327 catalog_manager.cc:5495] T b242bc596f1c4c03a6972ad0b32614b3 P 83902cee8ba9431989cb5e9a7e9f2696 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83902cee8ba9431989cb5e9a7e9f2696 (127.21.143.67). New cstate: current_term: 1 leader_uuid: "83902cee8ba9431989cb5e9a7e9f2696" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "863a0e5a8a294a52b82f829980953745" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 40213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "938d1fb62bf04e448806046383b864c0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 34801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83902cee8ba9431989cb5e9a7e9f2696" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 44159 } health_report { overall_health: HEALTHY } } }
W20260619 18:43:26.177809 321 tablet.cc:2369] T b242bc596f1c4c03a6972ad0b32614b3 P 83902cee8ba9431989cb5e9a7e9f2696: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:43:26.205915 336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:26.202384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39160 (local address 127.21.143.67:44159)
0619 18:43:26.202514 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:26.202519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:26.202535 (+ 16us) server_negotiation.cc:407] Connection header received
0619 18:43:26.202659 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:26.202661 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:26.202714 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:26.202816 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:43:26.203071 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:26.203638 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:26.204496 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:26.204728 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:26.204832 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:26.205127 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:43:26.205519 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:43:26.205647 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:26.205710 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 18:43:26.205763 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 18:43:26.211835 32608 raft_consensus.cc:1261] T b242bc596f1c4c03a6972ad0b32614b3 P 863a0e5a8a294a52b82f829980953745 [term 1 FOLLOWER]: Refusing update from remote peer 83902cee8ba9431989cb5e9a7e9f2696: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:43:26.211835 32474 raft_consensus.cc:1261] T b242bc596f1c4c03a6972ad0b32614b3 P 938d1fb62bf04e448806046383b864c0 [term 1 FOLLOWER]: Refusing update from remote peer 83902cee8ba9431989cb5e9a7e9f2696: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:43:26.212513 347 consensus_queue.cc:1035] T b242bc596f1c4c03a6972ad0b32614b3 P 83902cee8ba9431989cb5e9a7e9f2696 [LEADER]: Connected to new peer: Peer: permanent_uuid: "938d1fb62bf04e448806046383b864c0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 34801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:43:26.212646 345 consensus_queue.cc:1035] T b242bc596f1c4c03a6972ad0b32614b3 P 83902cee8ba9431989cb5e9a7e9f2696 [LEADER]: Connected to new peer: Peer: permanent_uuid: "863a0e5a8a294a52b82f829980953745" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 40213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:43:26.221316 351 mvcc.cc:205] Tried to move back new op lower bound from 7298640307036512256 to 7298640306817945600. Current Snapshot: MvccSnapshot[applied={T|T < 7298640307036512256}]
I20260619 18:43:26.223176 352 mvcc.cc:205] Tried to move back new op lower bound from 7298640307036512256 to 7298640306817945600. Current Snapshot: MvccSnapshot[applied={T|T < 7298640307036512256}]
I20260619 18:43:26.227439 32327 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:43:26.227563 32327 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:43:26.229619 32327 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c36be2383e242f1a8f35af9e843c13a: Sending DeleteTablet for 3 replicas of tablet b242bc596f1c4c03a6972ad0b32614b3
I20260619 18:43:26.230378 32722 tablet_service.cc:1514] Processing DeleteTablet for tablet b242bc596f1c4c03a6972ad0b32614b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:43:26 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:39152
I20260619 18:43:26.230489 32588 tablet_service.cc:1514] Processing DeleteTablet for tablet b242bc596f1c4c03a6972ad0b32614b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:43:26 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:44852
I20260619 18:43:26.230791 360 tablet_replica.cc:330] T b242bc596f1c4c03a6972ad0b32614b3 P 83902cee8ba9431989cb5e9a7e9f2696: stopping tablet replica
I20260619 18:43:26.230903 361 tablet_replica.cc:330] T b242bc596f1c4c03a6972ad0b32614b3 P 863a0e5a8a294a52b82f829980953745: stopping tablet replica
I20260619 18:43:26.230907 32454 tablet_service.cc:1514] Processing DeleteTablet for tablet b242bc596f1c4c03a6972ad0b32614b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:43:26 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:48650
I20260619 18:43:26.231070 360 raft_consensus.cc:2229] T b242bc596f1c4c03a6972ad0b32614b3 P 83902cee8ba9431989cb5e9a7e9f2696 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:43:26.231132 361 raft_consensus.cc:2229] T b242bc596f1c4c03a6972ad0b32614b3 P 863a0e5a8a294a52b82f829980953745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:43:26.231405 361 raft_consensus.cc:2258] T b242bc596f1c4c03a6972ad0b32614b3 P 863a0e5a8a294a52b82f829980953745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:43:26.231459 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 32391
I20260619 18:43:26.232561 360 raft_consensus.cc:2258] T b242bc596f1c4c03a6972ad0b32614b3 P 83902cee8ba9431989cb5e9a7e9f2696 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:43:26.233408 361 ts_tablet_manager.cc:1905] T b242bc596f1c4c03a6972ad0b32614b3 P 863a0e5a8a294a52b82f829980953745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:43:26.233877 360 ts_tablet_manager.cc:1905] T b242bc596f1c4c03a6972ad0b32614b3 P 83902cee8ba9431989cb5e9a7e9f2696: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:43:26.236130 361 ts_tablet_manager.cc:1918] T b242bc596f1c4c03a6972ad0b32614b3 P 863a0e5a8a294a52b82f829980953745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:43:26.236219 361 log.cc:1196] T b242bc596f1c4c03a6972ad0b32614b3 P 863a0e5a8a294a52b82f829980953745: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-1/wal/wals/b242bc596f1c4c03a6972ad0b32614b3
I20260619 18:43:26.236531 361 ts_tablet_manager.cc:1939] T b242bc596f1c4c03a6972ad0b32614b3 P 863a0e5a8a294a52b82f829980953745: Deleting consensus metadata
I20260619 18:43:26.236512 360 ts_tablet_manager.cc:1918] T b242bc596f1c4c03a6972ad0b32614b3 P 83902cee8ba9431989cb5e9a7e9f2696: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:43:26.236603 360 log.cc:1196] T b242bc596f1c4c03a6972ad0b32614b3 P 83902cee8ba9431989cb5e9a7e9f2696: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894542572984-22077-0/minicluster-data/ts-2/wal/wals/b242bc596f1c4c03a6972ad0b32614b3
I20260619 18:43:26.236908 360 ts_tablet_manager.cc:1939] T b242bc596f1c4c03a6972ad0b32614b3 P 83902cee8ba9431989cb5e9a7e9f2696: Deleting consensus metadata
I20260619 18:43:26.237706 32314 catalog_manager.cc:4841] TS 863a0e5a8a294a52b82f829980953745 (127.21.143.66:40213): tablet b242bc596f1c4c03a6972ad0b32614b3 (table test-table [id=afaef5fdc4e74a94a056e8a5a7630c9b]) successfully deleted
I20260619 18:43:26.237993 32314 catalog_manager.cc:4841] TS 83902cee8ba9431989cb5e9a7e9f2696 (127.21.143.67:44159): tablet b242bc596f1c4c03a6972ad0b32614b3 (table test-table [id=afaef5fdc4e74a94a056e8a5a7630c9b]) successfully deleted
W20260619 18:43:26.242725 32314 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.143.65:34801 (error 108)
I20260619 18:43:26.242986 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 32525
I20260619 18:43:26.246021 335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:26.244007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:34801 (local address 127.0.0.1:48656)
0619 18:43:26.245838 (+ 1831us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:26.245921 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.65:34801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1723}
W20260619 18:43:26.246197 32314 catalog_manager.cc:4584] TS 938d1fb62bf04e448806046383b864c0 (127.21.143.65:34801): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b242bc596f1c4c03a6972ad0b32614b3: Network error: Client connection negotiation failed: client connection to 127.21.143.65:34801: connect: Connection refused (error 111)
I20260619 18:43:26.249003 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 32659
I20260619 18:43:26.254777 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 32297
2026-06-19T18:43:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2343 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:43:26 dist-test-slave-mld5 krb5kdc[366](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:43:26 dist-test-slave-mld5 krb5kdc[366](info): set up 2 sockets
Jun 19 18:43:26 dist-test-slave-mld5 krb5kdc[366](info): commencing operation
krb5kdc: starting...
W20260619 18:43:27.292219 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:43:27 dist-test-slave-mld5 krb5kdc[366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:43:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:43:27Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.21.143.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.21.143.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.21.143.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/rangeradmin_127.21.143.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.21.143.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/rangeradmin_127.21.143.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.21.143.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.21.143.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.21.143.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/rangerlookup_127.21.143.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.21.143.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/rangerlookup_127.21.143.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.143.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.21.143.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.143.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/HTTP_127.21.143.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.143.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/HTTP_127.21.143.84@KRBTEST.COM.keytab.
I20260619 18:43:27.445041 22077 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task1WQx4B/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/postgres -l logfile start
2026-06-19 18:43:29.734 UTC [396] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 18:43:29.734 UTC [396] LOG: listening on IPv4 address "127.21.143.84", port 34951
2026-06-19 18:43:29.736 UTC [396] LOG: listening on Unix socket "/tmp/.s.PGSQL.34951"
2026-06-19 18:43:29.756 UTC [399] LOG: database system was shut down at 2026-06-19 18:43:28 UTC
2026-06-19 18:43:29.759 UTC [396] LOG: database system is ready to accept connections
I20260619 18:43:30.716037 22077 mini_postgres.cc:96] Postgres bound to 34951
2026-06-19 18:43:30.722 UTC [407] FATAL: database "slave" does not exist
127.21.143.84:34951 - accepting connections
I20260619 18:43:30.723083 22077 mini_ranger.cc:161] Starting Ranger...
I20260619 18:43:30.738706 22077 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 18:43:31.180580 22077 mini_ranger.cc:87] Created ranger Postgres database
I20260619 18:43:31.180655 22077 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ranger-admin
2026-06-19 18:43:31,429 [I] DB FLAVOR :POSTGRES
2026-06-19 18:43:31,430 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 18:43:31,430 [I] Checking connection..
2026-06-19 18:43:31,430 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:34951/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 18:43:31,664 [I] Checking connection passed.
2026-06-19 18:43:31,665 [I] --------- Verifying version history table ---------
2026-06-19 18:43:31,665 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:34951/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 18:43:31,913 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 18:43:31,913 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:34951/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 18:43:32,168 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 18:43:32,168 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 18:43:32,168 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:34951/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 18:43:32.416 UTC [471] WARNING: there is no transaction in progress
2026-06-19 18:43:32,422 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 18:43:32,423 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:34951/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 18:43:32,661 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 18:43:32,661 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 18:43:32,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:34951/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 18:43:32,885 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:34951/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 18:43:33,138 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:34951/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-mld5.c.gcp-upstream.internal','N') ;"
2026-06-19 18:43:33,372 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:34951/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 18:43:33,622 [I] Table x_portal_user does not exist in database ranger
2026-06-19 18:43:33,622 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:34951/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 18:43:33,872 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 18:43:33,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:34951/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 18:43:34,097 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:34951/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 18:43:34,354 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 18:43:34,354 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:34951/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 18:43:35.281 UTC [579] WARNING: there is no transaction in progress
2026-06-19 18:43:35.295 UTC [579] WARNING: there is no transaction in progress
2026-06-19 18:43:35.311 UTC [579] WARNING: there is no transaction in progress
2026-06-19 18:43:35.325 UTC [579] WARNING: there is no transaction in progress
2026-06-19 18:43:35.340 UTC [579] WARNING: there is no transaction in progress
2026-06-19 18:43:35.356 UTC [579] WARNING: there is no transaction in progress
2026-06-19 18:43:35.408 UTC [579] WARNING: there is no transaction in progress
2026-06-19 18:43:35.414 UTC [579] WARNING: there is no transaction in progress
2026-06-19 18:43:35.428 UTC [579] WARNING: there is no transaction in progress
2026-06-19 18:43:35.437 UTC [579] WARNING: there is no transaction in progress
2026-06-19 18:43:35.447 UTC [579] WARNING: there is no transaction in progress
2026-06-19 18:43:35.460 UTC [579] WARNING: there is no transaction in progress
2026-06-19 18:43:35.469 UTC [579] WARNING: there is no transaction in progress
2026-06-19 18:43:35.474 UTC [579] WARNING: there is no transaction in progress
2026-06-19 18:43:35.480 UTC [579] WARNING: there is no transaction in progress
2026-06-19 18:43:35.485 UTC [579] WARNING: there is no transaction in progress
2026-06-19 18:43:36,055 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 18:43:36,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:34951/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 18:43:36,298 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 18:43:36,299 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:34951/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 18:43:36,567 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 18:43:36,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:34951/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 18:43:36,822 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 18:43:36,823 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:34951/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 18:43:37,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:34951/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 18:43:37,307 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:34951/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-mld5.c.gcp-upstream.internal';"
2026-06-19 18:43:37,541 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 18:43:37,541 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:34951/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 18:43:37,775 [I] DB_PATCHES have already been applied
I20260619 18:43:37.781256 22077 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ranger-admin:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task1WQx4B/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 18:43:37.781313 22077 mini_ranger.cc:193] Using host: 127.21.143.84
I20260619 18:43:37.784785 22077 mini_ranger.cc:227] Ranger admin URL: http://127.21.143.84:35785
Jun 19, 2026 6:43:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:43:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:43:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 6:43:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 6:43:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 6:43:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 6:43:38 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 18:43:38.898293 22077 mini_ranger.cc:160] Time spent starting Ranger: real 8.175s user 0.005s sys 0.005s
Jun 19, 2026 6:43:57 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 6:43:57 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 6:43:57 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 6:43:57 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 6:43:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 6:43:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 6:43:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 6:43:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 6:43:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 6:43:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 6:43:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 6:43:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 6:43:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 6:43:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 6:43:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 6:43:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 6:43:57 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 6:43:58 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 18:43:59.221276 22077 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.21.143.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.21.143.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.21.143.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/rangerkms_127.21.143.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.21.143.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/rangerkms_127.21.143.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.143.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.21.143.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.143.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/HTTP_127.21.143.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.143.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/HTTP_127.21.143.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 18:43:59 dist-test-slave-mld5 krb5kdc[366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894639, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 18:43:59.300745 22077 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 18:43:59.309926 22077 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 18:44:00.114653 22077 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 18:44:00.114756 22077 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ranger-kms
I20260619 18:44:00.114861 22077 mini_ranger_kms.cc:227] Using postgres at 127.21.143.84:34951
2026-06-19 18:44:00,329 [I] DB FLAVOR :POSTGRES
2026-06-19 18:44:00,330 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 18:44:00,330 [I] Checking connection
2026-06-19 18:44:00,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:34951/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 18:44:00,609 [I] connection success
2026-06-19 18:44:00,610 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 18:44:00,610 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 18:44:00,610 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:34951/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 18:44:00,881 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 18:44:00,881 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 18:44:00,881 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 18:44:00,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:34951/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 18:44:01,191 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 18:44:01.651193 22077 mini_ranger_kms.cc:310] Created kms service
I20260619 18:44:01.832760 22077 mini_ranger_kms.cc:326] Created kudu user
I20260619 18:44:01.877014 22077 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 18:44:01 dist-test-slave-mld5 krb5kdc[366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894639, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.21.143.84@KRBTEST.COM
I20260619 18:44:02.259802 22077 mini_ranger_kms.cc:384] Added ranger policy
I20260619 18:44:02.259979 22077 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ranger-kms:/tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 18:44:02.260010 22077 mini_ranger_kms.cc:240] Using host: 127.21.143.84
I20260619 18:44:02.264331 22077 mini_ranger_kms.cc:276] Ranger KMS PID: 759
I20260619 18:44:02.264480 22077 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.21.143.84:42659
Jun 19, 2026 6:44:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:44:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:44:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 6:44:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 6:44:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 6:44:03 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 18:44:04.545759 22077 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.245s user 0.006s sys 0.192s
I20260619 18:44:04.545902 22077 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 18:44:04.546032 22077 mini_ranger_kms.cc:401] 127.21.143.84:42659/kms/v1/keys
Jun 19 18:44:04 dist-test-slave-mld5 krb5kdc[366](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781894644, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.143.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 18:44:05 dist-test-slave-mld5 krb5kdc[366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894644, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.143.84@KRBTEST.COM for HTTP/127.21.143.84@KRBTEST.COM
Jun 19 18:44:05 dist-test-slave-mld5 krb5kdc[366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894644, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.143.84@KRBTEST.COM for HTTP/127.21.143.84@KRBTEST.COM
Jun 19 18:44:07 dist-test-slave-mld5 krb5kdc[366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.21.143.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
I20260619 18:44:07.635929 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:42021
--webserver_interface=127.21.143.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:38873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:42021
--ranger_config_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.143.84:42659/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:44:07.749094 810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:07.749325 810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:07.749379 810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:07.753259 810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:44:07.753305 810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:07.753324 810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:07.753340 810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:44:07.753355 810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:44:07.758716 810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:38873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/master-0/wal
--ranger_kms_url=127.21.143.84:42659/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:42021
--ranger_config_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:42021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:44:07.759591 810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:07.760495 810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 18:44:07.768669 810 server_base.cc:1034] running on GCE node
W20260619 18:44:07.768685 818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:07.768719 816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:07.768719 815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:07.769491 810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:07.770524 810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:07.771718 810 hybrid_clock.cc:648] HybridClock initialized: now 1781894647771697 us; error 41 us; skew 500 ppm
Jun 19 18:44:07 dist-test-slave-mld5 krb5kdc[366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894647, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:07.777711 810 init.cc:377] Logged in from keytab as kudu/127.21.143.126@KRBTEST.COM (short username kudu)
I20260619 18:44:07.778604 810 webserver.cc:460] Webserver started at http://127.21.143.126:35317/ using document root <none> and password file <none>
I20260619 18:44:07.779153 810 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:07.779233 810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:44:07.779485 810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:44:07 dist-test-slave-mld5 krb5kdc[366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894647, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for HTTP/127.21.143.84@KRBTEST.COM
I20260619 18:44:07.827085 810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "d806940808e74c668cb55994888c9c22"
format_stamp: "Formatted at 2026-06-19 18:44:07 on dist-test-slave-mld5"
server_key: "13914cc74588fd04cd0e68081c6efaba"
server_key_iv: "c236b24da1526220ec0f06aed58f11b9"
server_key_version: "kuduclusterkey@0"
I20260619 18:44:07.827760 810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "d806940808e74c668cb55994888c9c22"
format_stamp: "Formatted at 2026-06-19 18:44:07 on dist-test-slave-mld5"
server_key: "13914cc74588fd04cd0e68081c6efaba"
server_key_iv: "c236b24da1526220ec0f06aed58f11b9"
server_key_version: "kuduclusterkey@0"
I20260619 18:44:07.831701 810 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:44:07.845583 825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:07.847004 810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 18:44:07.847146 810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "d806940808e74c668cb55994888c9c22"
format_stamp: "Formatted at 2026-06-19 18:44:07 on dist-test-slave-mld5"
server_key: "13914cc74588fd04cd0e68081c6efaba"
server_key_iv: "c236b24da1526220ec0f06aed58f11b9"
server_key_version: "kuduclusterkey@0"
I20260619 18:44:07.847276 810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:07.857780 810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:07.860651 810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:07.860838 810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:07.868988 810 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:42021
I20260619 18:44:07.869025 877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:42021 every 8 connection(s)
I20260619 18:44:07.869943 810 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:44:07.872782 878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:44:07.873071 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 810
I20260619 18:44:07.873183 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:44:07.873481 22077 external_mini_cluster.cc:1428] Setting key 39bb66ed6fa2d72ee72442223644d090
I20260619 18:44:07.880628 878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d806940808e74c668cb55994888c9c22: Bootstrap starting.
I20260619 18:44:07.883188 878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d806940808e74c668cb55994888c9c22: Neither blocks nor log segments found. Creating new log.
I20260619 18:44:07.884001 878 log.cc:824] T 00000000000000000000000000000000 P d806940808e74c668cb55994888c9c22: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:44:07 dist-test-slave-mld5 krb5kdc[366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:44:07.886765 878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d806940808e74c668cb55994888c9c22: No bootstrap required, opened a new log
I20260619 18:44:07.891165 878 raft_consensus.cc:348] T 00000000000000000000000000000000 P d806940808e74c668cb55994888c9c22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d806940808e74c668cb55994888c9c22" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42021 } }
I20260619 18:44:07.891386 881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:07.875388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52654 (local address 127.21.143.126:42021)
0619 18:44:07.875886 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:07.875906 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:07.875950 (+ 44us) server_negotiation.cc:407] Connection header received
0619 18:44:07.876903 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:07.876928 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:07.877347 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:07.877707 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:07.878596 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:07.880097 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:07.881317 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:07.882026 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:07.882185 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:07.885144 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:07.885149 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:07.885161 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:07.885185 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:07.887862 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:07.888352 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:07.888359 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:07.888364 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:07.888499 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:07.888794 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:07.888800 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:07.888802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:07.889265 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:07.889443 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:07.889733 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 18:44:07.890421 (+ 688us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":140,"threads_started":1}
I20260619 18:44:07.891449 878 raft_consensus.cc:374] T 00000000000000000000000000000000 P d806940808e74c668cb55994888c9c22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:44:07.891502 878 raft_consensus.cc:729] T 00000000000000000000000000000000 P d806940808e74c668cb55994888c9c22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d806940808e74c668cb55994888c9c22, State: Initialized, Role: FOLLOWER
I20260619 18:44:07.891994 878 consensus_queue.cc:260] T 00000000000000000000000000000000 P d806940808e74c668cb55994888c9c22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d806940808e74c668cb55994888c9c22" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42021 } }
I20260619 18:44:07.892143 878 raft_consensus.cc:388] T 00000000000000000000000000000000 P d806940808e74c668cb55994888c9c22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:44:07.892225 878 raft_consensus.cc:482] T 00000000000000000000000000000000 P d806940808e74c668cb55994888c9c22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:44:07.892375 878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d806940808e74c668cb55994888c9c22 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:44:07.893452 878 raft_consensus.cc:504] T 00000000000000000000000000000000 P d806940808e74c668cb55994888c9c22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d806940808e74c668cb55994888c9c22" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42021 } }
I20260619 18:44:07.893875 878 leader_election.cc:302] T 00000000000000000000000000000000 P d806940808e74c668cb55994888c9c22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d806940808e74c668cb55994888c9c22; no voters:
I20260619 18:44:07.894208 878 leader_election.cc:288] T 00000000000000000000000000000000 P d806940808e74c668cb55994888c9c22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:44:07.894271 883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d806940808e74c668cb55994888c9c22 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:44:07.894459 883 raft_consensus.cc:686] T 00000000000000000000000000000000 P d806940808e74c668cb55994888c9c22 [term 1 LEADER]: Becoming Leader. State: Replica: d806940808e74c668cb55994888c9c22, State: Running, Role: LEADER
I20260619 18:44:07.894814 883 consensus_queue.cc:237] T 00000000000000000000000000000000 P d806940808e74c668cb55994888c9c22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d806940808e74c668cb55994888c9c22" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42021 } }
I20260619 18:44:07.895453 878 sys_catalog.cc:564] T 00000000000000000000000000000000 P d806940808e74c668cb55994888c9c22 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:44:07.896245 883 sys_catalog.cc:455] T 00000000000000000000000000000000 P d806940808e74c668cb55994888c9c22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d806940808e74c668cb55994888c9c22. Latest consensus state: current_term: 1 leader_uuid: "d806940808e74c668cb55994888c9c22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d806940808e74c668cb55994888c9c22" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42021 } } }
I20260619 18:44:07.897034 883 sys_catalog.cc:458] T 00000000000000000000000000000000 P d806940808e74c668cb55994888c9c22 [sys.catalog]: This master's current role is: LEADER
I20260619 18:44:07.896921 884 sys_catalog.cc:455] T 00000000000000000000000000000000 P d806940808e74c668cb55994888c9c22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d806940808e74c668cb55994888c9c22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d806940808e74c668cb55994888c9c22" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42021 } } }
I20260619 18:44:07.897802 884 sys_catalog.cc:458] T 00000000000000000000000000000000 P d806940808e74c668cb55994888c9c22 [sys.catalog]: This master's current role is: LEADER
I20260619 18:44:07.898075 895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:44:07.898092 878 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 18:44:07.904343 895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:44:07.921288 895 catalog_manager.cc:1347] Generated new cluster ID: ced97548d3b84b4ca731a27d18d9ebce
I20260619 18:44:07.921365 895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:44:07.944144 895 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:44:07.945731 895 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:44:07.951323 895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d806940808e74c668cb55994888c9c22: Generated new TSK 0
I20260619 18:44:07.952294 895 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.143.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab.
Entry for principal kudu/127.21.143.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab.
I20260619 18:44:08.644146 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.143.84:42659/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.65:0
--local_ip_for_outbound_sockets=127.21.143.65
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:42021
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:38873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:44:08.766139 923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:08.766453 923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:08.766568 923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:08.771193 923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:08.771286 923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:08.771427 923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.65
I20260619 18:44:08.776947 923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:38873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.21.143.84:42659/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:42021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.65
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:44:08.778095 923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:08.779093 923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:44:08.788906 929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:08.788905 928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:08.789305 931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:08.789567 923 server_base.cc:1034] running on GCE node
I20260619 18:44:08.789932 923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:08.790650 923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:08.792249 923 hybrid_clock.cc:648] HybridClock initialized: now 1781894648792220 us; error 45 us; skew 500 ppm
Jun 19 18:44:08 dist-test-slave-mld5 krb5kdc[366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894648, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:08.795807 923 init.cc:377] Logged in from keytab as kudu/127.21.143.65@KRBTEST.COM (short username kudu)
I20260619 18:44:08.796972 923 webserver.cc:460] Webserver started at http://127.21.143.65:46835/ using document root <none> and password file <none>
I20260619 18:44:08.797765 923 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:08.797899 923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:44:08.798158 923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:44:08 dist-test-slave-mld5 krb5kdc[366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894648, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for HTTP/127.21.143.84@KRBTEST.COM
I20260619 18:44:08.811033 923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-0/data/instance:
uuid: "947c4023fe144bf0ad5e6c6cbf6be8cd"
format_stamp: "Formatted at 2026-06-19 18:44:08 on dist-test-slave-mld5"
server_key: "973390cd79ea570839854b0212fb655d"
server_key_iv: "d6758b2de84df08ec33c58f5aeffbfbd"
server_key_version: "kuduclusterkey@0"
I20260619 18:44:08.811889 923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance:
uuid: "947c4023fe144bf0ad5e6c6cbf6be8cd"
format_stamp: "Formatted at 2026-06-19 18:44:08 on dist-test-slave-mld5"
server_key: "973390cd79ea570839854b0212fb655d"
server_key_iv: "d6758b2de84df08ec33c58f5aeffbfbd"
server_key_version: "kuduclusterkey@0"
I20260619 18:44:08.817404 923 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 18:44:08.834398 940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:08.836114 923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 18:44:08.836232 923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-0/wal
uuid: "947c4023fe144bf0ad5e6c6cbf6be8cd"
format_stamp: "Formatted at 2026-06-19 18:44:08 on dist-test-slave-mld5"
server_key: "973390cd79ea570839854b0212fb655d"
server_key_iv: "d6758b2de84df08ec33c58f5aeffbfbd"
server_key_version: "kuduclusterkey@0"
I20260619 18:44:08.836351 923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:08.877933 923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:08.881809 923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:08.882076 923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:08.882863 923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:44:08.884029 923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:44:08.884148 923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:08.884291 923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:44:08.884377 923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:08.895785 923 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.65:39167
I20260619 18:44:08.897117 923 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
I20260619 18:44:08.898773 1053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.65:39167 every 8 connection(s)
I20260619 18:44:08.903144 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 923
I20260619 18:44:08.903262 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance
I20260619 18:44:08.903579 22077 external_mini_cluster.cc:1428] Setting key bd19bae753c07d2213af612838d14f77
Jun 19 18:44:08 dist-test-slave-mld5 krb5kdc[366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894648, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.143.66@KRBTEST.COM; defaulting to no policy
I20260619 18:44:08.918467 1056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:08.900512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42021 (local address 127.21.143.65:47239)
0619 18:44:08.901817 (+ 1305us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:08.901872 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:08.902932 (+ 1060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:08.903820 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:08.903831 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:08.904229 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:08.904933 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:08.904953 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:08.908063 (+ 3110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:08.908067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:08.909521 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:08.909532 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:08.910214 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:08.910218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:08.910427 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:08.911149 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:08.911169 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:08.912936 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:08.915241 (+ 2305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:08.915248 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:08.915265 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:08.915628 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:08.915957 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:08.915961 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:08.915963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:08.916151 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:08.916608 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:08.916614 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:08.916905 (+ 291us) client_negotiation.cc:769] Sending connection context
0619 18:44:08.917083 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 18:44:08.917393 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1082,"thread_start_us":139,"threads_started":1}
I20260619 18:44:08.919797 1057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:08.901619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:47239 (local address 127.21.143.126:42021)
0619 18:44:08.903379 (+ 1760us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:08.903390 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:08.903408 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:44:08.903462 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:08.903468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:08.903545 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:08.903645 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:08.906972 (+ 3327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:08.907903 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:08.909687 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:08.910036 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:08.910775 (+ 739us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:08.913124 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:08.913128 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:08.913131 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:08.913156 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:08.915105 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:08.915762 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:08.915766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:08.915768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:08.915855 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:08.916266 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:08.916268 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:08.916270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:08.916477 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:08.919428 (+ 2951us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:08.919512 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 18:44:08.919637 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1638,"thread_start_us":103,"threads_started":1}
I20260619 18:44:08.921057 1054 heartbeater.cc:347] Connected to a master server at 127.21.143.126:42021
I20260619 18:44:08.921284 1054 heartbeater.cc:464] Registering TS with master...
I20260619 18:44:08.921772 1054 heartbeater.cc:511] Master 127.21.143.126:42021 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.66@KRBTEST.COM" created.
I20260619 18:44:08.923540 842 ts_manager.cc:194] Registered new tserver with Master: 947c4023fe144bf0ad5e6c6cbf6be8cd (127.21.143.65:39167)
I20260619 18:44:08.925113 842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.65@KRBTEST.COM'} at 127.21.143.65:47239
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab.
Entry for principal kudu/127.21.143.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab.
I20260619 18:44:08.939045 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.143.84:42659/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.66:0
--local_ip_for_outbound_sockets=127.21.143.66
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:42021
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:38873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:44:08 dist-test-slave-mld5 krb5kdc[366](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781894648, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 18:44:09.090989 1062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:09.091286 1062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:09.091377 1062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:09.095587 1062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:09.095675 1062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:09.095779 1062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.66
I20260619 18:44:09.101588 1062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:38873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.21.143.84:42659/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:42021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.66
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:44:09.102865 1062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:09.104105 1062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:44:09.115351 1069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:09.115860 1062 server_base.cc:1034] running on GCE node
W20260619 18:44:09.115983 1071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:09.116169 1068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:09.116672 1062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:09.117348 1062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:09.118618 1062 hybrid_clock.cc:648] HybridClock initialized: now 1781894649118580 us; error 46 us; skew 500 ppm
Jun 19 18:44:09 dist-test-slave-mld5 krb5kdc[366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894649, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:09.121565 1062 init.cc:377] Logged in from keytab as kudu/127.21.143.66@KRBTEST.COM (short username kudu)
I20260619 18:44:09.122731 1062 webserver.cc:460] Webserver started at http://127.21.143.66:46285/ using document root <none> and password file <none>
I20260619 18:44:09.123556 1062 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:09.123708 1062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:44:09.124015 1062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:44:09 dist-test-slave-mld5 krb5kdc[366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894649, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for HTTP/127.21.143.84@KRBTEST.COM
I20260619 18:44:09.147643 1062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-1/data/instance:
uuid: "1defc9a6e3b14be48316025ceb2050b7"
format_stamp: "Formatted at 2026-06-19 18:44:09 on dist-test-slave-mld5"
server_key: "8d7eec6c3c6d85d6a53aa5e561958cdb"
server_key_iv: "654c3276c61c15418a7b0bc331f2a895"
server_key_version: "kuduclusterkey@0"
I20260619 18:44:09.148422 1062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance:
uuid: "1defc9a6e3b14be48316025ceb2050b7"
format_stamp: "Formatted at 2026-06-19 18:44:09 on dist-test-slave-mld5"
server_key: "8d7eec6c3c6d85d6a53aa5e561958cdb"
server_key_iv: "654c3276c61c15418a7b0bc331f2a895"
server_key_version: "kuduclusterkey@0"
I20260619 18:44:09.153857 1062 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 18:44:09.171495 1078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:09.173164 1062 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260619 18:44:09.173291 1062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-1/wal
uuid: "1defc9a6e3b14be48316025ceb2050b7"
format_stamp: "Formatted at 2026-06-19 18:44:09 on dist-test-slave-mld5"
server_key: "8d7eec6c3c6d85d6a53aa5e561958cdb"
server_key_iv: "654c3276c61c15418a7b0bc331f2a895"
server_key_version: "kuduclusterkey@0"
I20260619 18:44:09.173413 1062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:09.203845 1062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:09.208467 1062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:09.208767 1062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:09.209548 1062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:44:09.210632 1062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:44:09.210695 1062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:09.210748 1062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:44:09.210779 1062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:09.233028 1062 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.66:35595
I20260619 18:44:09.234323 1062 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
I20260619 18:44:09.235885 1191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.66:35595 every 8 connection(s)
I20260619 18:44:09.241255 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 1062
I20260619 18:44:09.241374 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance
I20260619 18:44:09.241688 22077 external_mini_cluster.cc:1428] Setting key a754c6461647affc8f108fcf4bbfa6f1
Jun 19 18:44:09 dist-test-slave-mld5 krb5kdc[366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894649, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:44:09.252261 1057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:09.239146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:60769 (local address 127.21.143.126:42021)
0619 18:44:09.239271 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:09.239278 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:09.239321 (+ 43us) server_negotiation.cc:407] Connection header received
0619 18:44:09.239972 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:09.239975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:09.240031 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:09.240133 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:09.241464 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:09.242315 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:09.243997 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:09.244341 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:09.244417 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:09.247469 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:09.247474 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:09.247477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:09.247505 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:09.249661 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:09.250292 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:09.250296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:09.250298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:09.250380 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:09.250771 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:09.250774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:09.250776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:09.250995 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:09.251076 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:09.251908 (+ 832us) server_negotiation.cc:299] Negotiation successful
0619 18:44:09.252054 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
WARNING: no policy specified for kudu/127.21.143.67@KRBTEST.COM; defaulting to no policy
I20260619 18:44:09.254300 1194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:09.238009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42021 (local address 127.21.143.66:60769)
0619 18:44:09.238622 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:09.238689 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:09.239695 (+ 1006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:09.240349 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:09.240358 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:09.240682 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:09.241301 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:09.241319 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:09.242467 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:09.242470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:09.243860 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:09.243871 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:09.244619 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:09.244623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:09.244829 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:09.245546 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:09.245566 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:09.247245 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:09.249805 (+ 2560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:09.249811 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:09.249830 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:09.250180 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:09.250480 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:09.250483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:09.250485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:09.250664 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:09.251145 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:09.251168 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:09.251533 (+ 365us) client_negotiation.cc:769] Sending connection context
0619 18:44:09.252824 (+ 1291us) client_negotiation.cc:240] Negotiation successful
0619 18:44:09.253193 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":146,"threads_started":1}
I20260619 18:44:09.255911 1192 heartbeater.cc:347] Connected to a master server at 127.21.143.126:42021
I20260619 18:44:09.256194 1192 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.67@KRBTEST.COM" created.
I20260619 18:44:09.256807 1192 heartbeater.cc:511] Master 127.21.143.126:42021 requested a full tablet report, sending...
I20260619 18:44:09.259824 842 ts_manager.cc:194] Registered new tserver with Master: 1defc9a6e3b14be48316025ceb2050b7 (127.21.143.66:35595)
I20260619 18:44:09.260658 842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.66@KRBTEST.COM'} at 127.21.143.66:60769
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab.
Entry for principal kudu/127.21.143.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab.
I20260619 18:44:09.295054 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.143.84:42659/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.67:0
--local_ip_for_outbound_sockets=127.21.143.67
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:42021
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:38873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:44:09 dist-test-slave-mld5 krb5kdc[366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894648, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for HTTP/127.21.143.84@KRBTEST.COM
W20260619 18:44:09.518211 1197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:09.518572 1197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:09.518635 1197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:09.524565 1197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:09.524663 1197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:09.524791 1197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.67
I20260619 18:44:09.532506 1197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:38873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.21.143.84:42659/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:42021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.67
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:44:09.533864 1197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:09.535123 1197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:44:09.558296 1205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:09.558442 1202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:09.558768 1203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:09.559917 1197 server_base.cc:1034] running on GCE node
I20260619 18:44:09.560355 1197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:09.561055 1197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:09.564349 1197 hybrid_clock.cc:648] HybridClock initialized: now 1781894649564271 us; error 84 us; skew 500 ppm
Jun 19 18:44:09 dist-test-slave-mld5 krb5kdc[366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894649, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:09.568207 1197 init.cc:377] Logged in from keytab as kudu/127.21.143.67@KRBTEST.COM (short username kudu)
I20260619 18:44:09.569192 1197 webserver.cc:460] Webserver started at http://127.21.143.67:34749/ using document root <none> and password file <none>
I20260619 18:44:09.569856 1197 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:09.569919 1197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:44:09.570135 1197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:44:09 dist-test-slave-mld5 krb5kdc[366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894649, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for HTTP/127.21.143.84@KRBTEST.COM
I20260619 18:44:09.595134 1197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-2/data/instance:
uuid: "36a2fa914dac4270b4651b7ffaea9819"
format_stamp: "Formatted at 2026-06-19 18:44:09 on dist-test-slave-mld5"
server_key: "035664e6976870508f0fedbfa293fca1"
server_key_iv: "2c525204e6faf59bd9bb0269f22774b5"
server_key_version: "kuduclusterkey@0"
I20260619 18:44:09.596105 1197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance:
uuid: "36a2fa914dac4270b4651b7ffaea9819"
format_stamp: "Formatted at 2026-06-19 18:44:09 on dist-test-slave-mld5"
server_key: "035664e6976870508f0fedbfa293fca1"
server_key_iv: "2c525204e6faf59bd9bb0269f22774b5"
server_key_version: "kuduclusterkey@0"
I20260619 18:44:09.601266 1197 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 18:44:09.616775 1212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:09.618947 1197 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260619 18:44:09.619197 1197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-2/wal
uuid: "36a2fa914dac4270b4651b7ffaea9819"
format_stamp: "Formatted at 2026-06-19 18:44:09 on dist-test-slave-mld5"
server_key: "035664e6976870508f0fedbfa293fca1"
server_key_iv: "2c525204e6faf59bd9bb0269f22774b5"
server_key_version: "kuduclusterkey@0"
I20260619 18:44:09.619483 1197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:09.645936 1197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:09.649724 1197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:09.649964 1197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:09.650720 1197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:44:09.651825 1197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:44:09.651932 1197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:09.652033 1197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:44:09.652094 1197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:09.665406 1197 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.67:33201
I20260619 18:44:09.666587 1197 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
I20260619 18:44:09.666860 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 1197
I20260619 18:44:09.666988 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance
I20260619 18:44:09.667376 22077 external_mini_cluster.cc:1428] Setting key 297c4eccbd425a7aa525c79588b9d68b
I20260619 18:44:09.668735 1325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.67:33201 every 8 connection(s)
Jun 19 18:44:09 dist-test-slave-mld5 krb5kdc[366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894649, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:44:09.686086 1057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:09.671970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:37539 (local address 127.21.143.126:42021)
0619 18:44:09.672083 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:09.672087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:09.672837 (+ 750us) server_negotiation.cc:407] Connection header received
0619 18:44:09.674029 (+ 1192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:09.674033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:09.674086 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:09.674136 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:09.675772 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:09.676654 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:09.678344 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:09.678717 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:09.678794 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:09.682010 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:09.682014 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:09.682016 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:09.682041 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:09.683811 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:09.684411 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:09.684415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:09.684417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:09.684498 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:09.684887 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:09.684891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:09.684893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:09.685093 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:09.685172 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:09.685821 (+ 649us) server_negotiation.cc:299] Negotiation successful
0619 18:44:09.685933 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 18:44:09.687820 1328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:09.670938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42021 (local address 127.21.143.67:37539)
0619 18:44:09.672698 (+ 1760us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:09.672750 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:09.673789 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:09.674549 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:09.674557 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:09.674899 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:09.675618 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:09.675636 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:09.676806 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:09.676810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:09.678182 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:09.678196 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:09.678992 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:09.678996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:09.679194 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:09.680080 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:09.680102 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:09.681837 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:09.683935 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:09.683941 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:09.683957 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:09.684283 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:09.684613 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:09.684616 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:09.684618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:09.684783 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:09.685234 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:09.685240 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:09.685536 (+ 296us) client_negotiation.cc:769] Sending connection context
0619 18:44:09.686484 (+ 948us) client_negotiation.cc:240] Negotiation successful
0619 18:44:09.686806 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1543,"thread_start_us":121,"threads_started":1}
I20260619 18:44:09.689173 1326 heartbeater.cc:347] Connected to a master server at 127.21.143.126:42021
I20260619 18:44:09.689389 1326 heartbeater.cc:464] Registering TS with master...
I20260619 18:44:09.689941 1326 heartbeater.cc:511] Master 127.21.143.126:42021 requested a full tablet report, sending...
I20260619 18:44:09.691028 842 ts_manager.cc:194] Registered new tserver with Master: 36a2fa914dac4270b4651b7ffaea9819 (127.21.143.67:33201)
I20260619 18:44:09.691798 842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.67@KRBTEST.COM'} at 127.21.143.67:37539
I20260619 18:44:09.694763 22077 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:44:09.716652 1057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:09.703850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52656 (local address 127.21.143.126:42021)
0619 18:44:09.703966 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:09.703971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:09.707627 (+ 3656us) server_negotiation.cc:407] Connection header received
0619 18:44:09.707823 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:09.707827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:09.707874 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:09.707926 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:09.708901 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:09.709706 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:09.710926 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:09.711251 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:09.711350 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:09.713167 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:09.713170 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:09.713173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:09.713200 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:09.714851 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:09.715342 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:09.715346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:09.715348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:09.715425 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:09.715722 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:09.715726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:09.715728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:09.715922 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:09.716019 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:09.716366 (+ 347us) server_negotiation.cc:299] Negotiation successful
0619 18:44:09.716478 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 18:44:09.720501 842 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:44:09.724984 842 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:44:09.748910 1339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:09.736437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:33201 (local address 127.0.0.1:34896)
0619 18:44:09.741267 (+ 4830us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:09.741322 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:09.741461 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:09.742205 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:09.742212 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:09.742234 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:44:09.742364 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:09.742369 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:09.744647 (+ 2278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:09.744651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:09.746064 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:09.746074 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:09.748148 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:09.748151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:09.748257 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:09.748523 (+ 266us) client_negotiation.cc:769] Sending connection context
0619 18:44:09.748615 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 18:44:09.748753 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3733,"mutex_wait_us":209,"thread_start_us":82,"threads_started":1}
I20260619 18:44:09.749337 1340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:09.737348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34896 (local address 127.21.143.67:33201)
0619 18:44:09.741234 (+ 3886us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:09.741245 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:09.741622 (+ 377us) server_negotiation.cc:407] Connection header received
0619 18:44:09.741702 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:09.741707 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:09.741917 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:09.742070 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:44:09.742561 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:09.744513 (+ 1952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:09.746219 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:09.747852 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:09.747951 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:09.748088 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:09.749107 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0619 18:44:09.749202 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3783,"thread_start_us":598,"threads_started":1}
I20260619 18:44:09.753532 1259 tablet_service.cc:1467] Processing CreateTablet for tablet 123dcc0eb3ab482eb5cf1dab4439ac34 (DEFAULT_TABLE table=test-table [id=33046ee5852e4c05a2ae8d65a2a573ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:44:09.754640 1259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 123dcc0eb3ab482eb5cf1dab4439ac34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:44:09.756969 1343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:09.739837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:35595 (local address 127.0.0.1:48722)
0619 18:44:09.743536 (+ 3699us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:09.743548 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:09.743623 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:09.748233 (+ 4610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:09.748236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:09.748253 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:44:09.748378 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:09.748384 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:09.750805 (+ 2421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:09.750809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:09.752020 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:09.752031 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:09.756547 (+ 4516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:09.756551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:09.756639 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:09.756663 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 18:44:09.756752 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 18:44:09.756832 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3647,"thread_start_us":78,"threads_started":1}
I20260619 18:44:09.757421 1344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:09.739994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48722 (local address 127.21.143.66:35595)
0619 18:44:09.747525 (+ 7531us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:09.747541 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:09.747568 (+ 27us) server_negotiation.cc:407] Connection header received
0619 18:44:09.747647 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:09.747652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:09.747889 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:09.748071 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:44:09.749493 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:09.750658 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:09.755370 (+ 4712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:09.756237 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:09.756343 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:09.756484 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:09.757144 (+ 660us) server_negotiation.cc:299] Negotiation successful
0619 18:44:09.757248 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7418,"thread_start_us":89,"threads_started":1}
I20260619 18:44:09.760365 1126 tablet_service.cc:1467] Processing CreateTablet for tablet 123dcc0eb3ab482eb5cf1dab4439ac34 (DEFAULT_TABLE table=test-table [id=33046ee5852e4c05a2ae8d65a2a573ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:44:09.761472 1126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 123dcc0eb3ab482eb5cf1dab4439ac34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:44:09.763863 1341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:09.739522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:39167 (local address 127.0.0.1:45404)
0619 18:44:09.743262 (+ 3740us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:09.743276 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:09.743373 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:09.748865 (+ 5492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:09.748868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:09.748884 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:44:09.748964 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:09.748969 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:09.760975 (+ 12006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:09.760979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:09.762228 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:09.762239 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:09.763365 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:09.763369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:09.763461 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:09.763486 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 18:44:09.763537 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 18:44:09.763599 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3674,"thread_start_us":87,"threads_started":1}
I20260619 18:44:09.764726 1342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:09.739659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45404 (local address 127.21.143.65:39167)
0619 18:44:09.748442 (+ 8783us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:09.748449 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:09.748470 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:44:09.748538 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:09.748542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:09.748726 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:09.755562 (+ 6836us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:44:09.755726 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:09.756783 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:09.762404 (+ 5621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:09.763204 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:09.764289 (+ 1085us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:09.764425 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:09.764481 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 18:44:09.764568 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8684,"thread_start_us":59,"threads_started":1}
I20260619 18:44:09.765530 1345 tablet_bootstrap.cc:492] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 36a2fa914dac4270b4651b7ffaea9819: Bootstrap starting.
I20260619 18:44:09.767619 988 tablet_service.cc:1467] Processing CreateTablet for tablet 123dcc0eb3ab482eb5cf1dab4439ac34 (DEFAULT_TABLE table=test-table [id=33046ee5852e4c05a2ae8d65a2a573ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:44:09.768684 988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 123dcc0eb3ab482eb5cf1dab4439ac34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:44:09.769423 1345 tablet_bootstrap.cc:654] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 36a2fa914dac4270b4651b7ffaea9819: Neither blocks nor log segments found. Creating new log.
I20260619 18:44:09.770275 1345 log.cc:824] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 36a2fa914dac4270b4651b7ffaea9819: Log is configured to *not* fsync() on all Append() calls
I20260619 18:44:09.772478 1345 tablet_bootstrap.cc:492] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 36a2fa914dac4270b4651b7ffaea9819: No bootstrap required, opened a new log
I20260619 18:44:09.772694 1345 ts_tablet_manager.cc:1397] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 36a2fa914dac4270b4651b7ffaea9819: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 18:44:09.778373 1345 raft_consensus.cc:348] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 36a2fa914dac4270b4651b7ffaea9819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a2fa914dac4270b4651b7ffaea9819" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33201 } } peers { permanent_uuid: "1defc9a6e3b14be48316025ceb2050b7" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35595 } } peers { permanent_uuid: "947c4023fe144bf0ad5e6c6cbf6be8cd" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 39167 } }
I20260619 18:44:09.778658 1345 raft_consensus.cc:374] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 36a2fa914dac4270b4651b7ffaea9819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:44:09.778715 1345 raft_consensus.cc:729] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 36a2fa914dac4270b4651b7ffaea9819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36a2fa914dac4270b4651b7ffaea9819, State: Initialized, Role: FOLLOWER
I20260619 18:44:09.779258 1345 consensus_queue.cc:260] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 36a2fa914dac4270b4651b7ffaea9819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a2fa914dac4270b4651b7ffaea9819" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33201 } } peers { permanent_uuid: "1defc9a6e3b14be48316025ceb2050b7" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35595 } } peers { permanent_uuid: "947c4023fe144bf0ad5e6c6cbf6be8cd" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 39167 } }
I20260619 18:44:09.780319 1345 ts_tablet_manager.cc:1428] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 36a2fa914dac4270b4651b7ffaea9819: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260619 18:44:09.780673 1326 heartbeater.cc:503] Master 127.21.143.126:42021 was elected leader, sending a full tablet report...
I20260619 18:44:09.786152 1349 tablet_bootstrap.cc:492] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 1defc9a6e3b14be48316025ceb2050b7: Bootstrap starting.
I20260619 18:44:09.789759 1349 tablet_bootstrap.cc:654] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 1defc9a6e3b14be48316025ceb2050b7: Neither blocks nor log segments found. Creating new log.
I20260619 18:44:09.790711 1349 log.cc:824] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 1defc9a6e3b14be48316025ceb2050b7: Log is configured to *not* fsync() on all Append() calls
I20260619 18:44:09.791484 1348 tablet_bootstrap.cc:492] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 947c4023fe144bf0ad5e6c6cbf6be8cd: Bootstrap starting.
I20260619 18:44:09.793744 1348 tablet_bootstrap.cc:654] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 947c4023fe144bf0ad5e6c6cbf6be8cd: Neither blocks nor log segments found. Creating new log.
I20260619 18:44:09.794569 1348 log.cc:824] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 947c4023fe144bf0ad5e6c6cbf6be8cd: Log is configured to *not* fsync() on all Append() calls
I20260619 18:44:09.797514 1349 tablet_bootstrap.cc:492] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 1defc9a6e3b14be48316025ceb2050b7: No bootstrap required, opened a new log
I20260619 18:44:09.797739 1349 ts_tablet_manager.cc:1397] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 1defc9a6e3b14be48316025ceb2050b7: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260619 18:44:09.806260 1349 raft_consensus.cc:348] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 1defc9a6e3b14be48316025ceb2050b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a2fa914dac4270b4651b7ffaea9819" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33201 } } peers { permanent_uuid: "1defc9a6e3b14be48316025ceb2050b7" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35595 } } peers { permanent_uuid: "947c4023fe144bf0ad5e6c6cbf6be8cd" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 39167 } }
I20260619 18:44:09.806571 1349 raft_consensus.cc:374] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 1defc9a6e3b14be48316025ceb2050b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:44:09.806630 1349 raft_consensus.cc:729] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 1defc9a6e3b14be48316025ceb2050b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1defc9a6e3b14be48316025ceb2050b7, State: Initialized, Role: FOLLOWER
I20260619 18:44:09.807196 1349 consensus_queue.cc:260] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 1defc9a6e3b14be48316025ceb2050b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a2fa914dac4270b4651b7ffaea9819" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33201 } } peers { permanent_uuid: "1defc9a6e3b14be48316025ceb2050b7" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35595 } } peers { permanent_uuid: "947c4023fe144bf0ad5e6c6cbf6be8cd" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 39167 } }
I20260619 18:44:09.807363 1348 tablet_bootstrap.cc:492] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 947c4023fe144bf0ad5e6c6cbf6be8cd: No bootstrap required, opened a new log
I20260619 18:44:09.807567 1348 ts_tablet_manager.cc:1397] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 947c4023fe144bf0ad5e6c6cbf6be8cd: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20260619 18:44:09.808290 1349 ts_tablet_manager.cc:1428] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 1defc9a6e3b14be48316025ceb2050b7: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260619 18:44:09.808645 1192 heartbeater.cc:503] Master 127.21.143.126:42021 was elected leader, sending a full tablet report...
I20260619 18:44:09.812544 1348 raft_consensus.cc:348] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 947c4023fe144bf0ad5e6c6cbf6be8cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a2fa914dac4270b4651b7ffaea9819" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33201 } } peers { permanent_uuid: "1defc9a6e3b14be48316025ceb2050b7" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35595 } } peers { permanent_uuid: "947c4023fe144bf0ad5e6c6cbf6be8cd" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 39167 } }
I20260619 18:44:09.812835 1348 raft_consensus.cc:374] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 947c4023fe144bf0ad5e6c6cbf6be8cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:44:09.812906 1348 raft_consensus.cc:729] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 947c4023fe144bf0ad5e6c6cbf6be8cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 947c4023fe144bf0ad5e6c6cbf6be8cd, State: Initialized, Role: FOLLOWER
I20260619 18:44:09.813439 1348 consensus_queue.cc:260] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 947c4023fe144bf0ad5e6c6cbf6be8cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a2fa914dac4270b4651b7ffaea9819" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33201 } } peers { permanent_uuid: "1defc9a6e3b14be48316025ceb2050b7" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35595 } } peers { permanent_uuid: "947c4023fe144bf0ad5e6c6cbf6be8cd" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 39167 } }
I20260619 18:44:09.814513 1348 ts_tablet_manager.cc:1428] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 947c4023fe144bf0ad5e6c6cbf6be8cd: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 18:44:09.814860 1054 heartbeater.cc:503] Master 127.21.143.126:42021 was elected leader, sending a full tablet report...
Jun 19 18:44:09 dist-test-slave-mld5 krb5kdc[366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894648, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for HTTP/127.21.143.84@KRBTEST.COM
W20260619 18:44:09.899019 1055 tablet.cc:2369] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 947c4023fe144bf0ad5e6c6cbf6be8cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:44:09.922669 1327 tablet.cc:2369] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 36a2fa914dac4270b4651b7ffaea9819: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:44:09.986485 1193 tablet.cc:2369] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 1defc9a6e3b14be48316025ceb2050b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:44:10.026274 1354 raft_consensus.cc:482] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 947c4023fe144bf0ad5e6c6cbf6be8cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:44:10.026525 1354 raft_consensus.cc:504] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 947c4023fe144bf0ad5e6c6cbf6be8cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a2fa914dac4270b4651b7ffaea9819" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33201 } } peers { permanent_uuid: "1defc9a6e3b14be48316025ceb2050b7" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35595 } } peers { permanent_uuid: "947c4023fe144bf0ad5e6c6cbf6be8cd" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 39167 } }
I20260619 18:44:10.028026 1354 leader_election.cc:288] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 947c4023fe144bf0ad5e6c6cbf6be8cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36a2fa914dac4270b4651b7ffaea9819 (127.21.143.67:33201), 1defc9a6e3b14be48316025ceb2050b7 (127.21.143.66:35595)
I20260619 18:44:10.035748 1340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:10.029184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:59967 (local address 127.21.143.67:33201)
0619 18:44:10.029460 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:10.029465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:10.029794 (+ 329us) server_negotiation.cc:407] Connection header received
0619 18:44:10.029998 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:10.030001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:10.030056 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:10.030127 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:44:10.030532 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:10.031357 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:10.033194 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:10.033812 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:10.033910 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:10.033965 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:10.035526 (+ 1561us) server_negotiation.cc:299] Negotiation successful
0619 18:44:10.035609 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194}
I20260619 18:44:10.036240 1356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:10.028423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:33201 (local address 127.21.143.65:59967)
0619 18:44:10.029699 (+ 1276us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:10.029715 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:10.029897 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:10.030280 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:10.030283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:10.030295 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:44:10.030413 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:10.030419 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:10.031504 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:10.031506 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:10.032898 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:10.032909 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:10.034962 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:10.034965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:10.035057 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:10.035082 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 18:44:10.035963 (+ 881us) client_negotiation.cc:240] Negotiation successful
0619 18:44:10.036048 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1203,"mutex_wait_us":31,"thread_start_us":113,"threads_started":1}
I20260619 18:44:10.036873 1344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:10.029321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:35307 (local address 127.21.143.66:35595)
0619 18:44:10.029543 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:10.029547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:10.030211 (+ 664us) server_negotiation.cc:407] Connection header received
0619 18:44:10.030401 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:10.030404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:10.030460 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:10.030512 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:44:10.030863 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:10.031581 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:10.034062 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:10.034708 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:10.034804 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:10.034852 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:10.036667 (+ 1815us) server_negotiation.cc:299] Negotiation successful
0619 18:44:10.036755 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260619 18:44:10.036939 1264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "123dcc0eb3ab482eb5cf1dab4439ac34" candidate_uuid: "947c4023fe144bf0ad5e6c6cbf6be8cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36a2fa914dac4270b4651b7ffaea9819" is_pre_election: true
I20260619 18:44:10.037307 1357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:10.028801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:35595 (local address 127.21.143.65:35307)
0619 18:44:10.030140 (+ 1339us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:10.030152 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:10.030287 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:10.030633 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:10.030636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:10.030651 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:44:10.030744 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:10.030750 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:10.031706 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:10.031709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:10.033053 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:10.033062 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:10.035234 (+ 2172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:10.035238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:10.035376 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:10.035400 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 18:44:10.037056 (+ 1656us) client_negotiation.cc:240] Negotiation successful
0619 18:44:10.037132 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1275,"mutex_wait_us":181,"thread_start_us":75,"threads_started":1}
I20260619 18:44:10.037314 1264 raft_consensus.cc:2454] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 36a2fa914dac4270b4651b7ffaea9819 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 947c4023fe144bf0ad5e6c6cbf6be8cd in term 0.
I20260619 18:44:10.037834 943 leader_election.cc:302] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 947c4023fe144bf0ad5e6c6cbf6be8cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36a2fa914dac4270b4651b7ffaea9819, 947c4023fe144bf0ad5e6c6cbf6be8cd; no voters:
I20260619 18:44:10.037892 1127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "123dcc0eb3ab482eb5cf1dab4439ac34" candidate_uuid: "947c4023fe144bf0ad5e6c6cbf6be8cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1defc9a6e3b14be48316025ceb2050b7" is_pre_election: true
I20260619 18:44:10.038161 1354 raft_consensus.cc:2781] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 947c4023fe144bf0ad5e6c6cbf6be8cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:44:10.038172 1127 raft_consensus.cc:2454] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 1defc9a6e3b14be48316025ceb2050b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 947c4023fe144bf0ad5e6c6cbf6be8cd in term 0.
I20260619 18:44:10.038249 1354 raft_consensus.cc:482] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 947c4023fe144bf0ad5e6c6cbf6be8cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:44:10.038291 1354 raft_consensus.cc:3037] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 947c4023fe144bf0ad5e6c6cbf6be8cd [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:44:10.039758 1354 raft_consensus.cc:504] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 947c4023fe144bf0ad5e6c6cbf6be8cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a2fa914dac4270b4651b7ffaea9819" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33201 } } peers { permanent_uuid: "1defc9a6e3b14be48316025ceb2050b7" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35595 } } peers { permanent_uuid: "947c4023fe144bf0ad5e6c6cbf6be8cd" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 39167 } }
I20260619 18:44:10.040119 1354 leader_election.cc:288] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 947c4023fe144bf0ad5e6c6cbf6be8cd [CANDIDATE]: Term 1 election: Requested vote from peers 36a2fa914dac4270b4651b7ffaea9819 (127.21.143.67:33201), 1defc9a6e3b14be48316025ceb2050b7 (127.21.143.66:35595)
I20260619 18:44:10.040688 1127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "123dcc0eb3ab482eb5cf1dab4439ac34" candidate_uuid: "947c4023fe144bf0ad5e6c6cbf6be8cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1defc9a6e3b14be48316025ceb2050b7"
I20260619 18:44:10.040831 1127 raft_consensus.cc:3037] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 1defc9a6e3b14be48316025ceb2050b7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:44:10.042222 1127 raft_consensus.cc:2454] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 1defc9a6e3b14be48316025ceb2050b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 947c4023fe144bf0ad5e6c6cbf6be8cd in term 1.
I20260619 18:44:10.042623 1264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "123dcc0eb3ab482eb5cf1dab4439ac34" candidate_uuid: "947c4023fe144bf0ad5e6c6cbf6be8cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36a2fa914dac4270b4651b7ffaea9819"
I20260619 18:44:10.042766 1264 raft_consensus.cc:3037] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 36a2fa914dac4270b4651b7ffaea9819 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:44:10.043432 943 leader_election.cc:302] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 947c4023fe144bf0ad5e6c6cbf6be8cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1defc9a6e3b14be48316025ceb2050b7, 947c4023fe144bf0ad5e6c6cbf6be8cd; no voters:
I20260619 18:44:10.044430 1264 raft_consensus.cc:2454] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 36a2fa914dac4270b4651b7ffaea9819 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 947c4023fe144bf0ad5e6c6cbf6be8cd in term 1.
I20260619 18:44:10.044891 1354 raft_consensus.cc:2781] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 947c4023fe144bf0ad5e6c6cbf6be8cd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:44:10.045125 1354 raft_consensus.cc:686] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 947c4023fe144bf0ad5e6c6cbf6be8cd [term 1 LEADER]: Becoming Leader. State: Replica: 947c4023fe144bf0ad5e6c6cbf6be8cd, State: Running, Role: LEADER
I20260619 18:44:10.045536 1354 consensus_queue.cc:237] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 947c4023fe144bf0ad5e6c6cbf6be8cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a2fa914dac4270b4651b7ffaea9819" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33201 } } peers { permanent_uuid: "1defc9a6e3b14be48316025ceb2050b7" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35595 } } peers { permanent_uuid: "947c4023fe144bf0ad5e6c6cbf6be8cd" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 39167 } }
I20260619 18:44:10.049391 842 catalog_manager.cc:5495] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 947c4023fe144bf0ad5e6c6cbf6be8cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 947c4023fe144bf0ad5e6c6cbf6be8cd (127.21.143.65). New cstate: current_term: 1 leader_uuid: "947c4023fe144bf0ad5e6c6cbf6be8cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a2fa914dac4270b4651b7ffaea9819" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1defc9a6e3b14be48316025ceb2050b7" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "947c4023fe144bf0ad5e6c6cbf6be8cd" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 39167 } health_report { overall_health: HEALTHY } } }
I20260619 18:44:10.111140 1342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:10.106996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45414 (local address 127.21.143.65:39167)
0619 18:44:10.107143 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:10.107147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:10.107164 (+ 17us) server_negotiation.cc:407] Connection header received
0619 18:44:10.107264 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:10.107267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:10.107362 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:10.107477 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:44:10.107703 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:10.108508 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:10.109369 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:10.109708 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:10.109825 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:10.110152 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:44:10.110740 (+ 588us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:44:10.110871 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:10.110910 (+ 39us) server_negotiation.cc:299] Negotiation successful
0619 18:44:10.110971 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 18:44:10.117314 1127 raft_consensus.cc:1261] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 1defc9a6e3b14be48316025ceb2050b7 [term 1 FOLLOWER]: Refusing update from remote peer 947c4023fe144bf0ad5e6c6cbf6be8cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:44:10.117327 1264 raft_consensus.cc:1261] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 36a2fa914dac4270b4651b7ffaea9819 [term 1 FOLLOWER]: Refusing update from remote peer 947c4023fe144bf0ad5e6c6cbf6be8cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:44:10.118065 1354 consensus_queue.cc:1035] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 947c4023fe144bf0ad5e6c6cbf6be8cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "36a2fa914dac4270b4651b7ffaea9819" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:44:10.118533 1358 consensus_queue.cc:1035] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 947c4023fe144bf0ad5e6c6cbf6be8cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1defc9a6e3b14be48316025ceb2050b7" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:44:10.127225 1369 mvcc.cc:205] Tried to move back new op lower bound from 7298640486872743936 to 7298640486589820928. Current Snapshot: MvccSnapshot[applied={T|T < 7298640486872743936}]
I20260619 18:44:10.127935 1368 mvcc.cc:205] Tried to move back new op lower bound from 7298640486872743936 to 7298640486589820928. Current Snapshot: MvccSnapshot[applied={T|T < 7298640486872743936}]
I20260619 18:44:10.130955 1370 mvcc.cc:205] Tried to move back new op lower bound from 7298640486872743936 to 7298640486589820928. Current Snapshot: MvccSnapshot[applied={T|T < 7298640486872743936}]
I20260619 18:44:10.141182 842 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:44:10.141392 842 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:44:10.144294 842 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d806940808e74c668cb55994888c9c22: Sending DeleteTablet for 3 replicas of tablet 123dcc0eb3ab482eb5cf1dab4439ac34
I20260619 18:44:10.145336 1259 tablet_service.cc:1514] Processing DeleteTablet for tablet 123dcc0eb3ab482eb5cf1dab4439ac34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:44:10 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:34896
I20260619 18:44:10.145915 1378 tablet_replica.cc:330] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 36a2fa914dac4270b4651b7ffaea9819: stopping tablet replica
I20260619 18:44:10.145890 988 tablet_service.cc:1514] Processing DeleteTablet for tablet 123dcc0eb3ab482eb5cf1dab4439ac34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:44:10 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:45404
I20260619 18:44:10.146389 1378 raft_consensus.cc:2229] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 36a2fa914dac4270b4651b7ffaea9819 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:44:10.146356 1126 tablet_service.cc:1514] Processing DeleteTablet for tablet 123dcc0eb3ab482eb5cf1dab4439ac34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:44:10 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:48722
I20260619 18:44:10.146824 1378 raft_consensus.cc:2258] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 36a2fa914dac4270b4651b7ffaea9819 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:44:10.147166 1379 tablet_replica.cc:330] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 947c4023fe144bf0ad5e6c6cbf6be8cd: stopping tablet replica
I20260619 18:44:10.147485 1379 raft_consensus.cc:2229] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 947c4023fe144bf0ad5e6c6cbf6be8cd [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:44:10.147596 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 923
I20260619 18:44:10.149096 1380 tablet_replica.cc:330] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 1defc9a6e3b14be48316025ceb2050b7: stopping tablet replica
I20260619 18:44:10.149163 1378 ts_tablet_manager.cc:1905] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 36a2fa914dac4270b4651b7ffaea9819: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:44:10.149441 1380 raft_consensus.cc:2229] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 1defc9a6e3b14be48316025ceb2050b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:44:10.149793 1380 raft_consensus.cc:2258] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 1defc9a6e3b14be48316025ceb2050b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:44:10.151449 1380 ts_tablet_manager.cc:1905] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 1defc9a6e3b14be48316025ceb2050b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:44:10.152915 1378 ts_tablet_manager.cc:1918] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 36a2fa914dac4270b4651b7ffaea9819: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:44:10.153069 1378 log.cc:1196] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 36a2fa914dac4270b4651b7ffaea9819: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-2/wal/wals/123dcc0eb3ab482eb5cf1dab4439ac34
I20260619 18:44:10.153478 1378 ts_tablet_manager.cc:1939] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 36a2fa914dac4270b4651b7ffaea9819: Deleting consensus metadata
I20260619 18:44:10.154942 1380 ts_tablet_manager.cc:1918] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 1defc9a6e3b14be48316025ceb2050b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:44:10.154927 828 catalog_manager.cc:4841] TS 36a2fa914dac4270b4651b7ffaea9819 (127.21.143.67:33201): tablet 123dcc0eb3ab482eb5cf1dab4439ac34 (table test-table [id=33046ee5852e4c05a2ae8d65a2a573ae]) successfully deleted
I20260619 18:44:10.155036 1380 log.cc:1196] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 1defc9a6e3b14be48316025ceb2050b7: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894542572984-22077-0/minicluster-data/ts-1/wal/wals/123dcc0eb3ab482eb5cf1dab4439ac34
I20260619 18:44:10.155390 1380 ts_tablet_manager.cc:1939] T 123dcc0eb3ab482eb5cf1dab4439ac34 P 1defc9a6e3b14be48316025ceb2050b7: Deleting consensus metadata
W20260619 18:44:10.155791 827 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.143.65:39167 (error 108)
I20260619 18:44:10.155992 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 1062
I20260619 18:44:10.157421 1341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:10.156999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:39167 (local address 127.0.0.1:45416)
0619 18:44:10.157241 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:10.157316 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.65:39167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260619 18:44:10.157596 827 catalog_manager.cc:4584] TS 947c4023fe144bf0ad5e6c6cbf6be8cd (127.21.143.65:39167): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 123dcc0eb3ab482eb5cf1dab4439ac34: Network error: Client connection negotiation failed: client connection to 127.21.143.65:39167: connect: Connection refused (error 111)
I20260619 18:44:10.161819 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 1197
I20260619 18:44:10.162670 1341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:10.161926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:35595 (local address 127.0.0.1:48728)
0619 18:44:10.162510 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:10.162573 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.66:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491}
I20260619 18:44:10.168774 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 810
I20260619 18:44:10.176810 22077 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 18:44:10.511629 22077 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 18:44:10.511787 22077 mini_ranger.cc:66] Stopping Ranger...
I20260619 18:44:10.856420 22077 mini_ranger.cc:68] Stopped Ranger
2026-06-19 18:44:10.856 UTC [396] LOG: received smart shutdown request
2026-06-19 18:44:10.860 UTC [396] LOG: background worker "logical replication launcher" (PID 405) exited with exit code 1
2026-06-19 18:44:10.860 UTC [400] LOG: shutting down
2026-06-19 18:44:11.003 UTC [396] LOG: database system is shut down
2026-06-19T18:44:11Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44865 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:44:11 dist-test-slave-mld5 krb5kdc[1390](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:44:11 dist-test-slave-mld5 krb5kdc[1390](info): set up 2 sockets
Jun 19 18:44:11 dist-test-slave-mld5 krb5kdc[1390](info): commencing operation
krb5kdc: starting...
W20260619 18:44:12.151475 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:44:12 dist-test-slave-mld5 krb5kdc[1390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:44:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:44:12Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.21.143.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.21.143.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.21.143.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/rangeradmin_127.21.143.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.21.143.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/rangeradmin_127.21.143.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.21.143.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.21.143.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.21.143.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/rangerlookup_127.21.143.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.21.143.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/rangerlookup_127.21.143.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.143.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.21.143.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.143.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/HTTP_127.21.143.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.143.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/HTTP_127.21.143.84@KRBTEST.COM.keytab.
I20260619 18:44:12.311847 22077 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task1WQx4B/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/postgres -l logfile start
2026-06-19 18:44:14.644 UTC [1420] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 18:44:14.644 UTC [1420] LOG: listening on IPv4 address "127.21.143.84", port 52549
2026-06-19 18:44:14.647 UTC [1420] LOG: listening on Unix socket "/tmp/.s.PGSQL.52549"
2026-06-19 18:44:14.668 UTC [1423] LOG: database system was shut down at 2026-06-19 18:44:13 UTC
2026-06-19 18:44:14.671 UTC [1420] LOG: database system is ready to accept connections
I20260619 18:44:15.639621 22077 mini_postgres.cc:96] Postgres bound to 52549
2026-06-19 18:44:15.645 UTC [1431] FATAL: database "slave" does not exist
127.21.143.84:52549 - accepting connections
I20260619 18:44:15.646313 22077 mini_ranger.cc:161] Starting Ranger...
I20260619 18:44:15.661327 22077 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 18:44:16.093266 22077 mini_ranger.cc:87] Created ranger Postgres database
I20260619 18:44:16.093365 22077 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ranger-admin
2026-06-19 18:44:16,368 [I] DB FLAVOR :POSTGRES
2026-06-19 18:44:16,369 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 18:44:16,369 [I] Checking connection..
2026-06-19 18:44:16,369 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:52549/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 18:44:16,634 [I] Checking connection passed.
2026-06-19 18:44:16,634 [I] --------- Verifying version history table ---------
2026-06-19 18:44:16,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:52549/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 18:44:16,891 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 18:44:16,892 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:52549/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 18:44:17,148 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 18:44:17,148 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 18:44:17,148 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:52549/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 18:44:17.382 UTC [1495] WARNING: there is no transaction in progress
2026-06-19 18:44:17,389 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 18:44:17,389 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:52549/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 18:44:17,649 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 18:44:17,649 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 18:44:17,649 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:52549/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 18:44:17,895 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:52549/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 18:44:18,150 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:52549/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-mld5.c.gcp-upstream.internal','N') ;"
2026-06-19 18:44:18,368 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:52549/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 18:44:18,631 [I] Table x_portal_user does not exist in database ranger
2026-06-19 18:44:18,631 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:52549/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 18:44:18,889 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 18:44:18,889 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:52549/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 18:44:19,114 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:52549/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 18:44:19,366 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 18:44:19,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:52549/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 18:44:20.289 UTC [1603] WARNING: there is no transaction in progress
2026-06-19 18:44:20.305 UTC [1603] WARNING: there is no transaction in progress
2026-06-19 18:44:20.320 UTC [1603] WARNING: there is no transaction in progress
2026-06-19 18:44:20.334 UTC [1603] WARNING: there is no transaction in progress
2026-06-19 18:44:20.350 UTC [1603] WARNING: there is no transaction in progress
2026-06-19 18:44:20.365 UTC [1603] WARNING: there is no transaction in progress
2026-06-19 18:44:20.417 UTC [1603] WARNING: there is no transaction in progress
2026-06-19 18:44:20.424 UTC [1603] WARNING: there is no transaction in progress
2026-06-19 18:44:20.437 UTC [1603] WARNING: there is no transaction in progress
2026-06-19 18:44:20.447 UTC [1603] WARNING: there is no transaction in progress
2026-06-19 18:44:20.457 UTC [1603] WARNING: there is no transaction in progress
2026-06-19 18:44:20.471 UTC [1603] WARNING: there is no transaction in progress
2026-06-19 18:44:20.480 UTC [1603] WARNING: there is no transaction in progress
2026-06-19 18:44:20.486 UTC [1603] WARNING: there is no transaction in progress
2026-06-19 18:44:20.492 UTC [1603] WARNING: there is no transaction in progress
2026-06-19 18:44:20.498 UTC [1603] WARNING: there is no transaction in progress
2026-06-19 18:44:21,085 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 18:44:21,086 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:52549/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 18:44:21,329 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 18:44:21,329 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:52549/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 18:44:21,590 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 18:44:21,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:52549/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 18:44:21,822 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 18:44:21,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:52549/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 18:44:22,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:52549/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 18:44:22,278 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:52549/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-mld5.c.gcp-upstream.internal';"
2026-06-19 18:44:22,528 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 18:44:22,528 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:52549/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 18:44:22,798 [I] DB_PATCHES have already been applied
I20260619 18:44:22.803937 22077 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ranger-admin:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task1WQx4B/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 18:44:22.804016 22077 mini_ranger.cc:193] Using host: 127.21.143.84
I20260619 18:44:22.807382 22077 mini_ranger.cc:227] Ranger admin URL: http://127.21.143.84:40981
Jun 19, 2026 6:44:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:44:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:44:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 6:44:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 6:44:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 6:44:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 6:44:23 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 18:44:23.911278 22077 mini_ranger.cc:160] Time spent starting Ranger: real 8.265s user 0.000s sys 0.010s
Jun 19, 2026 6:44:42 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 6:44:42 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 6:44:42 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 6:44:42 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 6:44:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 6:44:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 6:44:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 6:44:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 6:44:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 6:44:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 6:44:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 6:44:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 6:44:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 6:44:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 6:44:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 6:44:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 6:44:42 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 6:44:43 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 18:44:44.419335 22077 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.21.143.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.21.143.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.21.143.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/rangerkms_127.21.143.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.21.143.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/rangerkms_127.21.143.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.143.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.21.143.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.143.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/HTTP_127.21.143.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.143.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/HTTP_127.21.143.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 18:44:44 dist-test-slave-mld5 krb5kdc[1390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894684, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 18:44:44.502563 22077 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 18:44:44.511785 22077 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 18:44:45.289902 22077 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 18:44:45.289987 22077 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ranger-kms
I20260619 18:44:45.290011 22077 mini_ranger_kms.cc:227] Using postgres at 127.21.143.84:52549
2026-06-19 18:44:45,459 [I] DB FLAVOR :POSTGRES
2026-06-19 18:44:45,459 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 18:44:45,459 [I] Checking connection
2026-06-19 18:44:45,459 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:52549/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 18:44:45,693 [I] connection success
2026-06-19 18:44:45,693 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 18:44:45,693 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 18:44:45,693 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:52549/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 18:44:45,931 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 18:44:45,931 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 18:44:45,932 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 18:44:45,932 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.143.84:52549/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 18:44:46,182 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 18:44:46.651823 22077 mini_ranger_kms.cc:310] Created kms service
I20260619 18:44:46.835520 22077 mini_ranger_kms.cc:326] Created kudu user
I20260619 18:44:46.880653 22077 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 18:44:46 dist-test-slave-mld5 krb5kdc[1390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894684, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.21.143.84@KRBTEST.COM
I20260619 18:44:47.272541 22077 mini_ranger_kms.cc:384] Added ranger policy
I20260619 18:44:47.272696 22077 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ranger-kms:/tmp/dist-test-task1WQx4B/build/debug/bin/postgresql.jar:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task1WQx4B/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 18:44:47.272719 22077 mini_ranger_kms.cc:240] Using host: 127.21.143.84
I20260619 18:44:47.276588 22077 mini_ranger_kms.cc:276] Ranger KMS PID: 1783
I20260619 18:44:47.276688 22077 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.21.143.84:33345
Jun 19, 2026 6:44:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:44:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:44:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 6:44:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 6:44:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 6:44:48 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 19 18:44:49 dist-test-slave-mld5 krb5kdc[1390](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781894689, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.143.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:49.537840 22077 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.035s user 0.011s sys 0.142s
I20260619 18:44:49.537945 22077 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 18:44:49.538024 22077 mini_ranger_kms.cc:401] 127.21.143.84:33345/kms/v1/keys
Jun 19 18:44:49 dist-test-slave-mld5 krb5kdc[1390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894689, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.143.84@KRBTEST.COM for HTTP/127.21.143.84@KRBTEST.COM
Jun 19 18:44:50 dist-test-slave-mld5 krb5kdc[1390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894689, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.143.84@KRBTEST.COM for HTTP/127.21.143.84@KRBTEST.COM
Jun 19 18:44:52 dist-test-slave-mld5 krb5kdc[1390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.21.143.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
I20260619 18:44:52.406255 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:42297
--webserver_interface=127.21.143.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:42673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:42297
--ranger_config_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.143.84:33345/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:44:52.547955 1834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:52.548209 1834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:52.548247 1834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:52.552107 1834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:44:52.552158 1834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:52.552177 1834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:52.552193 1834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:44:52.552208 1834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:44:52.557647 1834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:42673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/master-0/wal
--ranger_kms_url=127.21.143.84:33345/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:42297
--ranger_config_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:42297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:44:52.558717 1834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:52.561023 1834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:44:52.573251 1839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:52.573221 1840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:52.573210 1842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:52.573948 1834 server_base.cc:1034] running on GCE node
I20260619 18:44:52.574782 1834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:52.576399 1834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:52.577630 1834 hybrid_clock.cc:648] HybridClock initialized: now 1781894692577592 us; error 56 us; skew 500 ppm
Jun 19 18:44:52 dist-test-slave-mld5 krb5kdc[1390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894692, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:52.581306 1834 init.cc:377] Logged in from keytab as kudu/127.21.143.126@KRBTEST.COM (short username kudu)
I20260619 18:44:52.582262 1834 webserver.cc:460] Webserver started at http://127.21.143.126:36817/ using document root <none> and password file <none>
I20260619 18:44:52.582856 1834 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:52.582935 1834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:44:52.583148 1834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:44:52 dist-test-slave-mld5 krb5kdc[1390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894692, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for HTTP/127.21.143.84@KRBTEST.COM
I20260619 18:44:52.631721 1834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "21de688ffc8049088c10cb7d07da7210"
format_stamp: "Formatted at 2026-06-19 18:44:52 on dist-test-slave-mld5"
server_key: "229cb62c0234f023aa3f25e97d73c009"
server_key_iv: "22486ccd1646403c8436ec7030d17458"
server_key_version: "kuduclusterkey@0"
I20260619 18:44:52.632483 1834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "21de688ffc8049088c10cb7d07da7210"
format_stamp: "Formatted at 2026-06-19 18:44:52 on dist-test-slave-mld5"
server_key: "229cb62c0234f023aa3f25e97d73c009"
server_key_iv: "22486ccd1646403c8436ec7030d17458"
server_key_version: "kuduclusterkey@0"
I20260619 18:44:52.637697 1834 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 18:44:52.657302 1849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:52.659171 1834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:44:52.659370 1834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "21de688ffc8049088c10cb7d07da7210"
format_stamp: "Formatted at 2026-06-19 18:44:52 on dist-test-slave-mld5"
server_key: "229cb62c0234f023aa3f25e97d73c009"
server_key_iv: "22486ccd1646403c8436ec7030d17458"
server_key_version: "kuduclusterkey@0"
I20260619 18:44:52.659528 1834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:52.686875 1834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:52.690272 1834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:52.690485 1834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:52.698846 1834 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:42297
I20260619 18:44:52.699023 1901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:42297 every 8 connection(s)
I20260619 18:44:52.700030 1834 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:44:52.703855 1902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:44:52.704855 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 1834
I20260619 18:44:52.704991 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:44:52.705310 22077 external_mini_cluster.cc:1428] Setting key 08b69c06281eda0980150fc35759ea23
I20260619 18:44:52.711246 1902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210: Bootstrap starting.
Jun 19 18:44:52 dist-test-slave-mld5 krb5kdc[1390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:44:52.714145 1902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210: Neither blocks nor log segments found. Creating new log.
I20260619 18:44:52.715126 1902 log.cc:824] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210: Log is configured to *not* fsync() on all Append() calls
I20260619 18:44:52.717592 1902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210: No bootstrap required, opened a new log
I20260619 18:44:52.720084 1905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:52.706793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43340 (local address 127.21.143.126:42297)
0619 18:44:52.707233 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:52.707248 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:52.707283 (+ 35us) server_negotiation.cc:407] Connection header received
0619 18:44:52.707970 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:52.707986 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:52.708362 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:52.708747 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:52.709536 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:52.710501 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:52.711461 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:52.711925 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:52.712099 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:52.714615 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:52.714620 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:52.714635 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:52.714660 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:52.717075 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:52.717524 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:52.717534 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:52.717539 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:52.717665 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:52.717994 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:52.717999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:52.718000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:52.718450 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:52.718647 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:52.718935 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 18:44:52.719239 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":145,"threads_started":1}
I20260619 18:44:52.721827 1902 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21de688ffc8049088c10cb7d07da7210" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42297 } }
I20260619 18:44:52.722138 1902 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:44:52.722196 1902 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21de688ffc8049088c10cb7d07da7210, State: Initialized, Role: FOLLOWER
I20260619 18:44:52.722710 1902 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21de688ffc8049088c10cb7d07da7210" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42297 } }
I20260619 18:44:52.722889 1902 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:44:52.722957 1902 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:44:52.723057 1902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:44:52.724324 1902 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21de688ffc8049088c10cb7d07da7210" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42297 } }
I20260619 18:44:52.724758 1902 leader_election.cc:302] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21de688ffc8049088c10cb7d07da7210; no voters:
I20260619 18:44:52.725154 1902 leader_election.cc:288] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:44:52.725284 1907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:44:52.725539 1907 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210 [term 1 LEADER]: Becoming Leader. State: Replica: 21de688ffc8049088c10cb7d07da7210, State: Running, Role: LEADER
I20260619 18:44:52.725875 1907 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21de688ffc8049088c10cb7d07da7210" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42297 } }
I20260619 18:44:52.726577 1902 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:44:52.727622 1909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21de688ffc8049088c10cb7d07da7210" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21de688ffc8049088c10cb7d07da7210" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42297 } } }
I20260619 18:44:52.727769 1909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210 [sys.catalog]: This master's current role is: LEADER
I20260619 18:44:52.727990 1909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21de688ffc8049088c10cb7d07da7210. Latest consensus state: current_term: 1 leader_uuid: "21de688ffc8049088c10cb7d07da7210" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21de688ffc8049088c10cb7d07da7210" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42297 } } }
I20260619 18:44:52.728077 1909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210 [sys.catalog]: This master's current role is: LEADER
I20260619 18:44:52.728869 1902 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 18:44:52.730293 1916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:44:52.736236 1916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:44:52.745289 1916 catalog_manager.cc:1347] Generated new cluster ID: a5ffc834c4884f84a6856a57408e319b
I20260619 18:44:52.745373 1916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:44:52.771703 1916 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:44:52.773957 1916 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:44:52.785096 1916 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210: Generated new TSK 0
I20260619 18:44:52.785979 1916 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.143.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab.
Entry for principal kudu/127.21.143.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab.
I20260619 18:44:53.504441 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.143.84:33345/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.65:0
--local_ip_for_outbound_sockets=127.21.143.65
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:42297
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:42673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:44:53.624078 1947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:53.624325 1947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:53.624364 1947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:53.628605 1947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:53.628656 1947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:53.628731 1947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.65
I20260619 18:44:53.634145 1947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:42673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.21.143.84:33345/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:42297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.65
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:44:53.635192 1947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:53.636258 1947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:44:53.646337 1953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:53.646355 1955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:53.646703 1952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:53.646873 1947 server_base.cc:1034] running on GCE node
I20260619 18:44:53.647274 1947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:53.647951 1947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:53.649132 1947 hybrid_clock.cc:648] HybridClock initialized: now 1781894693649103 us; error 48 us; skew 500 ppm
Jun 19 18:44:53 dist-test-slave-mld5 krb5kdc[1390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894693, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:53.652112 1947 init.cc:377] Logged in from keytab as kudu/127.21.143.65@KRBTEST.COM (short username kudu)
I20260619 18:44:53.653208 1947 webserver.cc:460] Webserver started at http://127.21.143.65:45699/ using document root <none> and password file <none>
I20260619 18:44:53.653899 1947 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:53.653986 1947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:44:53.654202 1947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:44:53 dist-test-slave-mld5 krb5kdc[1390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894693, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for HTTP/127.21.143.84@KRBTEST.COM
I20260619 18:44:53.669358 1947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-0/data/instance:
uuid: "0feef268011f4c35b898548bbd85d842"
format_stamp: "Formatted at 2026-06-19 18:44:53 on dist-test-slave-mld5"
server_key: "93399a9a24cc75b48a7d9ec7e44171db"
server_key_iv: "f5e425ccba112294af832c03ee06a11a"
server_key_version: "kuduclusterkey@0"
I20260619 18:44:53.670056 1947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance:
uuid: "0feef268011f4c35b898548bbd85d842"
format_stamp: "Formatted at 2026-06-19 18:44:53 on dist-test-slave-mld5"
server_key: "93399a9a24cc75b48a7d9ec7e44171db"
server_key_iv: "f5e425ccba112294af832c03ee06a11a"
server_key_version: "kuduclusterkey@0"
I20260619 18:44:53.676337 1947 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260619 18:44:53.701701 1964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:53.703713 1947 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20260619 18:44:53.703917 1947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-0/wal
uuid: "0feef268011f4c35b898548bbd85d842"
format_stamp: "Formatted at 2026-06-19 18:44:53 on dist-test-slave-mld5"
server_key: "93399a9a24cc75b48a7d9ec7e44171db"
server_key_iv: "f5e425ccba112294af832c03ee06a11a"
server_key_version: "kuduclusterkey@0"
I20260619 18:44:53.704149 1947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:53.723212 1947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:53.727043 1947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:53.727288 1947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:53.728169 1947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:44:53.729257 1947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:44:53.729321 1947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:53.729386 1947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:44:53.729463 1947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260619 18:44:53.739758 1947 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.65:37153
I20260619 18:44:53.739974 2077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.65:37153 every 8 connection(s)
I20260619 18:44:53.741101 1947 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
I20260619 18:44:53.742741 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 1947
I20260619 18:44:53.742866 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance
I20260619 18:44:53.743201 22077 external_mini_cluster.cc:1428] Setting key b913b0b00ee65f9ea057b4edce6b5bf1
Jun 19 18:44:53 dist-test-slave-mld5 krb5kdc[1390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894693, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.143.66@KRBTEST.COM; defaulting to no policy
I20260619 18:44:53.758303 2080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:53.743634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:40937 (local address 127.21.143.126:42297)
0619 18:44:53.744077 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:53.744086 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:53.744924 (+ 838us) server_negotiation.cc:407] Connection header received
0619 18:44:53.746109 (+ 1185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:53.746112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:53.746165 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:53.746224 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:53.747894 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:53.748503 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:53.750226 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:53.750601 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:53.750719 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:53.753815 (+ 3096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:53.753820 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:53.753823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:53.753851 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:53.755760 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:53.756412 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:53.756416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:53.756417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:53.756506 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:53.756903 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:53.756907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:53.756909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:53.757160 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:53.757234 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:53.757960 (+ 726us) server_negotiation.cc:299] Negotiation successful
0619 18:44:53.758104 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":97,"threads_started":1}
I20260619 18:44:53.760262 2081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:53.744191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42297 (local address 127.21.143.65:40937)
0619 18:44:53.744785 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:53.744842 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:53.745862 (+ 1020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:53.746635 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:53.746644 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:53.746991 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:53.747730 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:53.747747 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:53.748667 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:53.748671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:53.750097 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:53.750108 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:53.750901 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:53.750905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:53.751113 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:53.751828 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:53.751849 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:53.753607 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:53.755900 (+ 2293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:53.755907 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:53.755936 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:53.756297 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:53.756610 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:53.756612 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:53.756614 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:53.756791 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:53.757318 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:53.757326 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:53.757660 (+ 334us) client_negotiation.cc:769] Sending connection context
0619 18:44:53.758773 (+ 1113us) client_negotiation.cc:240] Negotiation successful
0619 18:44:53.759147 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":136,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.66@KRBTEST.COM" created.
I20260619 18:44:53.766158 2078 heartbeater.cc:347] Connected to a master server at 127.21.143.126:42297
I20260619 18:44:53.766441 2078 heartbeater.cc:464] Registering TS with master...
I20260619 18:44:53.767088 2078 heartbeater.cc:511] Master 127.21.143.126:42297 requested a full tablet report, sending...
I20260619 18:44:53.768926 1866 ts_manager.cc:194] Registered new tserver with Master: 0feef268011f4c35b898548bbd85d842 (127.21.143.65:37153)
I20260619 18:44:53.770509 1866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.65@KRBTEST.COM'} at 127.21.143.65:40937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab.
Entry for principal kudu/127.21.143.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab.
I20260619 18:44:53.794181 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.143.84:33345/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.66:0
--local_ip_for_outbound_sockets=127.21.143.66
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:42297
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:42673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:44:53 dist-test-slave-mld5 krb5kdc[1390](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781894693, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 18:44:53.959749 2086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:53.960155 2086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:53.960218 2086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:53.966743 2086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:53.966840 2086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:53.966938 2086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.66
I20260619 18:44:53.975129 2086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:42673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.21.143.84:33345/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:42297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.66
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:44:53.976722 2086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:53.978108 2086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:44:53.990661 2093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:53.990698 2092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:53.991343 2086 server_base.cc:1034] running on GCE node
W20260619 18:44:53.991351 2095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:53.991859 2086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:53.992741 2086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:53.993964 2086 hybrid_clock.cc:648] HybridClock initialized: now 1781894693993942 us; error 41 us; skew 500 ppm
Jun 19 18:44:53 dist-test-slave-mld5 krb5kdc[1390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894693, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:53.997768 2086 init.cc:377] Logged in from keytab as kudu/127.21.143.66@KRBTEST.COM (short username kudu)
I20260619 18:44:53.998896 2086 webserver.cc:460] Webserver started at http://127.21.143.66:44941/ using document root <none> and password file <none>
I20260619 18:44:53.999668 2086 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:53.999789 2086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:44:54.000059 2086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:44:53 dist-test-slave-mld5 krb5kdc[1390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894693, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for HTTP/127.21.143.84@KRBTEST.COM
I20260619 18:44:54.016474 2086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-1/data/instance:
uuid: "17f2fb1d1a6a4f3ea9abaada1d9e2d59"
format_stamp: "Formatted at 2026-06-19 18:44:54 on dist-test-slave-mld5"
server_key: "bc00535fdca47d5e50109f9a2503e634"
server_key_iv: "572d62b28bddb87ff96dafaf6700ca3e"
server_key_version: "kuduclusterkey@0"
I20260619 18:44:54.017606 2086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance:
uuid: "17f2fb1d1a6a4f3ea9abaada1d9e2d59"
format_stamp: "Formatted at 2026-06-19 18:44:54 on dist-test-slave-mld5"
server_key: "bc00535fdca47d5e50109f9a2503e634"
server_key_iv: "572d62b28bddb87ff96dafaf6700ca3e"
server_key_version: "kuduclusterkey@0"
I20260619 18:44:54.031620 2086 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.001s sys 0.004s
I20260619 18:44:54.056357 2102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:54.059132 2086 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 18:44:54.059332 2086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-1/wal
uuid: "17f2fb1d1a6a4f3ea9abaada1d9e2d59"
format_stamp: "Formatted at 2026-06-19 18:44:54 on dist-test-slave-mld5"
server_key: "bc00535fdca47d5e50109f9a2503e634"
server_key_iv: "572d62b28bddb87ff96dafaf6700ca3e"
server_key_version: "kuduclusterkey@0"
I20260619 18:44:54.059491 2086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:54.081086 2086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:54.084864 2086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:54.085110 2086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:54.085879 2086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:44:54.086957 2086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:44:54.087054 2086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:54.087138 2086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:44:54.087198 2086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:54.103053 2086 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.66:45009
I20260619 18:44:54.103111 2215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.66:45009 every 8 connection(s)
I20260619 18:44:54.104144 2086 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
I20260619 18:44:54.112870 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 2086
I20260619 18:44:54.112991 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance
I20260619 18:44:54.113327 22077 external_mini_cluster.cc:1428] Setting key 962a7975f68e57747a3ab5b00f29cc1e
Jun 19 18:44:54 dist-test-slave-mld5 krb5kdc[1390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894693, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.143.67@KRBTEST.COM; defaulting to no policy
I20260619 18:44:54.125425 2080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:54.106233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:51235 (local address 127.21.143.126:42297)
0619 18:44:54.106376 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:54.106381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:54.107717 (+ 1336us) server_negotiation.cc:407] Connection header received
0619 18:44:54.109000 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:54.109004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:54.109057 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:54.109115 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:54.110548 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:54.111108 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:54.112481 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:54.112811 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:54.112923 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:54.120328 (+ 7405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:54.120334 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:54.120340 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:54.120374 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:54.122394 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:54.123159 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:54.123163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:54.123164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:54.123274 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:54.123953 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:54.123958 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:54.123960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:54.124214 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:54.124312 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:54.125075 (+ 763us) server_negotiation.cc:299] Negotiation successful
0619 18:44:54.125226 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 18:44:54.126197 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:54.106699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42297 (local address 127.21.143.66:51235)
0619 18:44:54.107548 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:54.107608 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:54.108748 (+ 1140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:54.109550 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:54.109563 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:54.109847 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:54.110323 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:54.110336 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:54.111236 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:54.111241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:54.112340 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:54.112351 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:54.113758 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:54.113763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:54.114732 (+ 969us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:54.115464 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:54.115488 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:54.120103 (+ 4615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:54.122574 (+ 2471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:54.122581 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:54.122602 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:54.123004 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:54.123491 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:54.123496 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:54.123500 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:54.123822 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:54.124336 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:54.124343 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:54.124766 (+ 423us) client_negotiation.cc:769] Sending connection context
0619 18:44:54.124978 (+ 212us) client_negotiation.cc:240] Negotiation successful
0619 18:44:54.125252 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":153,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.67@KRBTEST.COM" created.
I20260619 18:44:54.127660 2216 heartbeater.cc:347] Connected to a master server at 127.21.143.126:42297
I20260619 18:44:54.127919 2216 heartbeater.cc:464] Registering TS with master...
I20260619 18:44:54.128527 2216 heartbeater.cc:511] Master 127.21.143.126:42297 requested a full tablet report, sending...
I20260619 18:44:54.129750 1866 ts_manager.cc:194] Registered new tserver with Master: 17f2fb1d1a6a4f3ea9abaada1d9e2d59 (127.21.143.66:45009)
I20260619 18:44:54.130427 1866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.66@KRBTEST.COM'} at 127.21.143.66:51235
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab.
Entry for principal kudu/127.21.143.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab.
I20260619 18:44:54.142323 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.143.84:33345/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.67:0
--local_ip_for_outbound_sockets=127.21.143.67
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:42297
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:42673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:44:54.306811 2221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:54.307238 2221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:54.307725 2221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:54.314363 2221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:54.314451 2221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:54.314549 2221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.67
Jun 19 18:44:54 dist-test-slave-mld5 krb5kdc[1390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894693, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for HTTP/127.21.143.84@KRBTEST.COM
I20260619 18:44:54.322953 2221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:42673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.21.143.84:33345/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:42297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.67
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:44:54.324425 2221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:54.325824 2221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:44:54.337455 2229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:54.338085 2227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:54.338706 2221 server_base.cc:1034] running on GCE node
W20260619 18:44:54.340628 2226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:54.341099 2221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:54.341804 2221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:54.345248 2221 hybrid_clock.cc:648] HybridClock initialized: now 1781894694344983 us; error 667 us; skew 500 ppm
Jun 19 18:44:54 dist-test-slave-mld5 krb5kdc[1390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894694, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:54.350659 2221 init.cc:377] Logged in from keytab as kudu/127.21.143.67@KRBTEST.COM (short username kudu)
I20260619 18:44:54.351835 2221 webserver.cc:460] Webserver started at http://127.21.143.67:37187/ using document root <none> and password file <none>
I20260619 18:44:54.352573 2221 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:54.352669 2221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:44:54.352909 2221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:44:54 dist-test-slave-mld5 krb5kdc[1390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894694, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for HTTP/127.21.143.84@KRBTEST.COM
I20260619 18:44:54.378913 2221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-2/data/instance:
uuid: "9db7e67ae5ef406b9111606b55924c8f"
format_stamp: "Formatted at 2026-06-19 18:44:54 on dist-test-slave-mld5"
server_key: "cd00481e2bbbeba6cf1b00829dd7167d"
server_key_iv: "5332bedecc0c09036ce8cf618727d68c"
server_key_version: "kuduclusterkey@0"
I20260619 18:44:54.392565 2221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance:
uuid: "9db7e67ae5ef406b9111606b55924c8f"
format_stamp: "Formatted at 2026-06-19 18:44:54 on dist-test-slave-mld5"
server_key: "cd00481e2bbbeba6cf1b00829dd7167d"
server_key_iv: "5332bedecc0c09036ce8cf618727d68c"
server_key_version: "kuduclusterkey@0"
I20260619 18:44:54.398548 2221 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260619 18:44:54.439324 2236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:54.441334 2221 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260619 18:44:54.441556 2221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-2/wal
uuid: "9db7e67ae5ef406b9111606b55924c8f"
format_stamp: "Formatted at 2026-06-19 18:44:54 on dist-test-slave-mld5"
server_key: "cd00481e2bbbeba6cf1b00829dd7167d"
server_key_iv: "5332bedecc0c09036ce8cf618727d68c"
server_key_version: "kuduclusterkey@0"
I20260619 18:44:54.441803 2221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:54.473436 2221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:54.476825 2221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:54.477031 2221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:54.477686 2221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:44:54.478621 2221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:44:54.478690 2221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:54.478756 2221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:44:54.478803 2221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:54.491250 2221 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.67:41433
I20260619 18:44:54.492621 2221 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
I20260619 18:44:54.496385 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 2221
I20260619 18:44:54.496510 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance
I20260619 18:44:54.496846 22077 external_mini_cluster.cc:1428] Setting key e72a62340191c18ce5312aa8b7fd3c57
I20260619 18:44:54.493379 2349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.67:41433 every 8 connection(s)
Jun 19 18:44:54 dist-test-slave-mld5 krb5kdc[1390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894694, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:44:54.518514 2080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:54.497771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:58605 (local address 127.21.143.126:42297)
0619 18:44:54.497889 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:54.497893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:54.503825 (+ 5932us) server_negotiation.cc:407] Connection header received
0619 18:44:54.505041 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:54.505045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:54.505101 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:54.505154 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:54.506991 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:54.507834 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:54.511257 (+ 3423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:54.511611 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:54.511685 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:54.514641 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:54.514645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:54.514648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:54.514675 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:54.516305 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:54.516901 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:54.516905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:54.516907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:54.516984 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:54.517336 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:54.517339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:54.517341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:54.517533 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:54.517608 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:54.518243 (+ 635us) server_negotiation.cc:299] Negotiation successful
0619 18:44:54.518360 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 18:44:54.520316 2352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:54.496713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42297 (local address 127.21.143.67:58605)
0619 18:44:54.503664 (+ 6951us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:54.503731 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:54.504791 (+ 1060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:54.505610 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:54.505620 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:54.506021 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:54.506808 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:54.506831 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:54.507982 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:54.507985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:54.511109 (+ 3124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:54.511120 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:54.511882 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:54.511886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:54.512085 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:54.512764 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:54.512785 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:54.514443 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:54.516433 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:54.516439 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:54.516455 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:54.516788 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:54.517082 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:54.517085 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:54.517087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:54.517244 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:54.517680 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:54.517686 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:54.517981 (+ 295us) client_negotiation.cc:769] Sending connection context
0619 18:44:54.518938 (+ 957us) client_negotiation.cc:240] Negotiation successful
0619 18:44:54.519264 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6690,"thread_start_us":172,"threads_started":1}
I20260619 18:44:54.522605 2350 heartbeater.cc:347] Connected to a master server at 127.21.143.126:42297
I20260619 18:44:54.522840 2350 heartbeater.cc:464] Registering TS with master...
I20260619 18:44:54.523403 2350 heartbeater.cc:511] Master 127.21.143.126:42297 requested a full tablet report, sending...
I20260619 18:44:54.524981 1866 ts_manager.cc:194] Registered new tserver with Master: 9db7e67ae5ef406b9111606b55924c8f (127.21.143.67:41433)
I20260619 18:44:54.525705 1866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.67@KRBTEST.COM'} at 127.21.143.67:58605
I20260619 18:44:54.536103 22077 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:44:54.536927 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 1947
I20260619 18:44:54.545934 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 2086
I20260619 18:44:54.554299 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 2221
I20260619 18:44:54.568632 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 1834
I20260619 18:44:54.575672 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:42297
--webserver_interface=127.21.143.126
--webserver_port=36817
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:42673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:42297
--ranger_config_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.143.84:33345/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.143.84:33345/kms with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:44:54.686360 2353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:54.686640 2353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:54.686724 2353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:54.690795 2353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:44:54.690871 2353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:54.690905 2353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:54.690963 2353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:44:54.691000 2353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:44:54.696458 2353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:42673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.143.84:33345/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:42297
--ranger_config_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:42297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=36817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:44:54.697556 2353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:54.698619 2353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:44:54.707223 2361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:54.707223 2358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:54.707492 2353 server_base.cc:1034] running on GCE node
W20260619 18:44:54.707274 2359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:54.708181 2353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:54.709304 2353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:54.710482 2353 hybrid_clock.cc:648] HybridClock initialized: now 1781894694710452 us; error 54 us; skew 500 ppm
Jun 19 18:44:54 dist-test-slave-mld5 krb5kdc[1390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894694, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:54.713248 2353 init.cc:377] Logged in from keytab as kudu/127.21.143.126@KRBTEST.COM (short username kudu)
I20260619 18:44:54.714253 2353 webserver.cc:460] Webserver started at http://127.21.143.126:36817/ using document root <none> and password file <none>
I20260619 18:44:54.714788 2353 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:54.714856 2353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 18:44:54 dist-test-slave-mld5 krb5kdc[1390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894694, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for HTTP/127.21.143.84@KRBTEST.COM
I20260619 18:44:54.761291 2353 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260619 18:44:54.763725 2369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:54.765122 2353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:44:54.765228 2353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "21de688ffc8049088c10cb7d07da7210"
format_stamp: "Formatted at 2026-06-19 18:44:52 on dist-test-slave-mld5"
server_key: "229cb62c0234f023aa3f25e97d73c009"
server_key_iv: "22486ccd1646403c8436ec7030d17458"
server_key_version: "kuduclusterkey@0"
I20260619 18:44:54.765826 2353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:54.786824 2353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:54.789841 2353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:54.790051 2353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:54.798102 2353 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:42297
I20260619 18:44:54.798115 2421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:42297 every 8 connection(s)
I20260619 18:44:54.799216 2353 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:44:54.802693 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 2353
I20260619 18:44:54.803188 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.143.84:33345/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.65:37153
--local_ip_for_outbound_sockets=127.21.143.65
--tserver_master_addrs=127.21.143.126:42297
--webserver_port=45699
--webserver_interface=127.21.143.65
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:42673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.143.84:33345/kms with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:44:54.803731 2422 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:44:54.805792 2422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210: Bootstrap starting.
I20260619 18:44:54.829430 2422 log.cc:824] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210: Log is configured to *not* fsync() on all Append() calls
I20260619 18:44:54.838588 2422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:44:54.839107 2422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210: Bootstrap complete.
I20260619 18:44:54.844538 2422 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21de688ffc8049088c10cb7d07da7210" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42297 } }
I20260619 18:44:54.845022 2422 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21de688ffc8049088c10cb7d07da7210, State: Initialized, Role: FOLLOWER
I20260619 18:44:54.845638 2422 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21de688ffc8049088c10cb7d07da7210" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42297 } }
I20260619 18:44:54.845780 2422 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:44:54.845842 2422 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:44:54.845927 2422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210 [term 1 FOLLOWER]: Advancing to term 2
I20260619 18:44:54.848527 2422 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21de688ffc8049088c10cb7d07da7210" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42297 } }
I20260619 18:44:54.848951 2422 leader_election.cc:302] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21de688ffc8049088c10cb7d07da7210; no voters:
I20260619 18:44:54.849316 2422 leader_election.cc:288] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 18:44:54.849525 2426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 18:44:54.849810 2426 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210 [term 2 LEADER]: Becoming Leader. State: Replica: 21de688ffc8049088c10cb7d07da7210, State: Running, Role: LEADER
I20260619 18:44:54.850179 2426 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21de688ffc8049088c10cb7d07da7210" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42297 } }
I20260619 18:44:54.850651 2422 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:44:54.851254 2426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "21de688ffc8049088c10cb7d07da7210" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21de688ffc8049088c10cb7d07da7210" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42297 } } }
I20260619 18:44:54.851456 2426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210 [sys.catalog]: This master's current role is: LEADER
I20260619 18:44:54.851688 2426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21de688ffc8049088c10cb7d07da7210. Latest consensus state: current_term: 2 leader_uuid: "21de688ffc8049088c10cb7d07da7210" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21de688ffc8049088c10cb7d07da7210" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42297 } } }
I20260619 18:44:54.851811 2426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210 [sys.catalog]: This master's current role is: LEADER
I20260619 18:44:54.852911 2422 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 18:44:54.853492 2433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:44:54.859462 2433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:44:54.860160 2433 catalog_manager.cc:1259] Loaded cluster ID: a5ffc834c4884f84a6856a57408e319b
I20260619 18:44:54.860222 2433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:44:54.862716 2433 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:44:54.863426 2433 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210: Loaded TSK: 0
I20260619 18:44:54.864714 2433 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 18:44:54.960717 2424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:54.961097 2424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:54.961166 2424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:54.967052 2424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:54.967128 2424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:54.967213 2424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.65
I20260619 18:44:54.974898 2424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:42673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.143.84:33345/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.65:37153
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.143.65
--webserver_port=45699
--tserver_master_addrs=127.21.143.126:42297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.65
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:44:54.976255 2424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:54.977560 2424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:44:54.989705 2462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:54.989728 2459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:54.989737 2460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:54.991353 2424 server_base.cc:1034] running on GCE node
I20260619 18:44:54.991861 2424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:54.992720 2424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:55.000169 2424 hybrid_clock.cc:648] HybridClock initialized: now 1781894695000141 us; error 51 us; skew 500 ppm
Jun 19 18:44:55 dist-test-slave-mld5 krb5kdc[1390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894694, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:55.003938 2424 init.cc:377] Logged in from keytab as kudu/127.21.143.65@KRBTEST.COM (short username kudu)
I20260619 18:44:55.005077 2424 webserver.cc:460] Webserver started at http://127.21.143.65:45699/ using document root <none> and password file <none>
I20260619 18:44:55.005790 2424 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:55.005899 2424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 18:44:55 dist-test-slave-mld5 krb5kdc[1390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894694, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for HTTP/127.21.143.84@KRBTEST.COM
I20260619 18:44:55.030990 2424 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260619 18:44:55.040266 2470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:55.052951 2424 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.003s sys 0.000s
I20260619 18:44:55.053153 2424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-0/wal
uuid: "0feef268011f4c35b898548bbd85d842"
format_stamp: "Formatted at 2026-06-19 18:44:53 on dist-test-slave-mld5"
server_key: "93399a9a24cc75b48a7d9ec7e44171db"
server_key_iv: "f5e425ccba112294af832c03ee06a11a"
server_key_version: "kuduclusterkey@0"
I20260619 18:44:55.053966 2424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:55.098865 2424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:55.102551 2424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:55.102759 2424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:55.103513 2424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:44:55.104547 2424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:44:55.104599 2424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:55.104650 2424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:44:55.104671 2424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:55.117069 2424 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.65:37153
I20260619 18:44:55.118470 2424 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
I20260619 18:44:55.120352 2583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.65:37153 every 8 connection(s)
I20260619 18:44:55.125058 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 2424
I20260619 18:44:55.125574 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.143.84:33345/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.66:45009
--local_ip_for_outbound_sockets=127.21.143.66
--tserver_master_addrs=127.21.143.126:42297
--webserver_port=44941
--webserver_interface=127.21.143.66
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:42673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.143.84:33345/kms with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:44:55 dist-test-slave-mld5 krb5kdc[1390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894694, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:44:55.145036 2586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:55.122811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42297 (local address 127.21.143.65:56713)
0619 18:44:55.123563 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:55.123621 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:55.124699 (+ 1078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:55.126464 (+ 1765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:55.126476 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:55.126871 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:55.127538 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:55.127558 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:55.129435 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:55.129438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:55.130859 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:55.130871 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:55.132162 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:55.132166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:55.132358 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:55.133110 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:55.133131 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:55.138294 (+ 5163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:55.141100 (+ 2806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:55.141107 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:55.141123 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:55.141493 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:55.141930 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:55.141933 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:55.141935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:55.142114 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:55.142953 (+ 839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:55.142961 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:55.143284 (+ 323us) client_negotiation.cc:769] Sending connection context
0619 18:44:55.143515 (+ 231us) client_negotiation.cc:240] Negotiation successful
0619 18:44:55.143899 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":148,"threads_started":1}
I20260619 18:44:55.147004 2587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:55.123449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:56713 (local address 127.21.143.126:42297)
0619 18:44:55.125210 (+ 1761us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:55.125232 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:55.125275 (+ 43us) server_negotiation.cc:407] Connection header received
0619 18:44:55.125558 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:55.125572 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:55.125992 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:55.126350 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:55.127751 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:55.129268 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:55.131008 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:55.131761 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:55.131928 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:55.138591 (+ 6663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:55.138598 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:55.138619 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:55.138660 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:55.140961 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:55.141641 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:55.141647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:55.141654 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:55.141817 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:55.142234 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:55.142237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:55.142239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:55.142761 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:55.142895 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:55.145714 (+ 2819us) server_negotiation.cc:299] Negotiation successful
0619 18:44:55.146040 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1605,"thread_start_us":120,"threads_started":1}
I20260619 18:44:55.149899 2584 heartbeater.cc:347] Connected to a master server at 127.21.143.126:42297
I20260619 18:44:55.150169 2584 heartbeater.cc:464] Registering TS with master...
I20260619 18:44:55.150846 2584 heartbeater.cc:511] Master 127.21.143.126:42297 requested a full tablet report, sending...
W20260619 18:44:55.153403 2584 heartbeater.cc:644] Failed to heartbeat to 127.21.143.126:42297 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 18:44:55.153542 2584 heartbeater.cc:464] Registering TS with master...
I20260619 18:44:55.153755 2584 heartbeater.cc:511] Master 127.21.143.126:42297 requested a full tablet report, sending...
I20260619 18:44:55.154726 2584 heartbeater.cc:464] Registering TS with master...
I20260619 18:44:55.154940 2584 heartbeater.cc:511] Master 127.21.143.126:42297 requested a full tablet report, sending...
W20260619 18:44:55.155745 2584 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 18:44:55.302206 2588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:55.302977 2588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:55.303112 2588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:55.309368 2588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:55.309558 2588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:55.309698 2588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.66
I20260619 18:44:55.317989 2588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:42673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.143.84:33345/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.66:45009
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.143.66
--webserver_port=44941
--tserver_master_addrs=127.21.143.126:42297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.66
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:44:55.319718 2588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:55.321117 2588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:44:55.333455 2594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:55.333578 2593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:55.333472 2596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:55.335248 2588 server_base.cc:1034] running on GCE node
I20260619 18:44:55.335767 2588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:55.336522 2588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:55.339762 2588 hybrid_clock.cc:648] HybridClock initialized: now 1781894695339727 us; error 56 us; skew 500 ppm
Jun 19 18:44:55 dist-test-slave-mld5 krb5kdc[1390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894695, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:55.343235 2588 init.cc:377] Logged in from keytab as kudu/127.21.143.66@KRBTEST.COM (short username kudu)
I20260619 18:44:55.344283 2588 webserver.cc:460] Webserver started at http://127.21.143.66:44941/ using document root <none> and password file <none>
I20260619 18:44:55.344959 2588 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:55.345059 2588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 18:44:55 dist-test-slave-mld5 krb5kdc[1390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894695, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for HTTP/127.21.143.84@KRBTEST.COM
I20260619 18:44:55.368942 2588 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.000s sys 0.003s
I20260619 18:44:55.373009 2604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:55.374694 2588 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 18:44:55.374820 2588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-1/wal
uuid: "17f2fb1d1a6a4f3ea9abaada1d9e2d59"
format_stamp: "Formatted at 2026-06-19 18:44:54 on dist-test-slave-mld5"
server_key: "bc00535fdca47d5e50109f9a2503e634"
server_key_iv: "572d62b28bddb87ff96dafaf6700ca3e"
server_key_version: "kuduclusterkey@0"
I20260619 18:44:55.376036 2588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:55.436445 2588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:55.440272 2588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:55.440470 2588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:55.441164 2588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:44:55.442160 2588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:44:55.442220 2588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:55.442278 2588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:44:55.442308 2588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:55.452977 2588 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.66:45009
I20260619 18:44:55.453054 2718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.66:45009 every 8 connection(s)
I20260619 18:44:55.454257 2588 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
I20260619 18:44:55.463781 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 2588
I20260619 18:44:55.464304 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.143.84:33345/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.67:41433
--local_ip_for_outbound_sockets=127.21.143.67
--tserver_master_addrs=127.21.143.126:42297
--webserver_port=37187
--webserver_interface=127.21.143.67
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:42673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.143.84:33345/kms with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:44:55 dist-test-slave-mld5 krb5kdc[1390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894695, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:44:55.470955 2587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:55.458103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:57625 (local address 127.21.143.126:42297)
0619 18:44:55.458256 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:55.458261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:55.458405 (+ 144us) server_negotiation.cc:407] Connection header received
0619 18:44:55.459726 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:55.459729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:55.459783 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:55.459835 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:55.461487 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:55.462269 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:55.463925 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:55.464245 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:55.464319 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:55.467053 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:55.467057 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:55.467059 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:55.467086 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:55.468714 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:55.469291 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:55.469294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:55.469296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:55.469375 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:55.469737 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:55.469740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:55.469742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:55.469941 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:55.470006 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:55.470659 (+ 653us) server_negotiation.cc:299] Negotiation successful
0619 18:44:55.470788 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 18:44:55.472803 2721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:55.456998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42297 (local address 127.21.143.66:57625)
0619 18:44:55.458246 (+ 1248us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:55.458302 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:55.459486 (+ 1184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:55.460265 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:55.460274 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:55.460672 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:55.461322 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:55.461341 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:55.462424 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:55.462427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:55.463794 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:55.463805 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:55.464514 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:55.464517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:55.464707 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:55.465397 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:55.465416 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:55.466872 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:55.468838 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:55.468844 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:55.468859 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:55.469180 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:55.469475 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:55.469478 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:55.469479 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:55.469636 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:55.470081 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:55.470087 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:55.470384 (+ 297us) client_negotiation.cc:769] Sending connection context
0619 18:44:55.471497 (+ 1113us) client_negotiation.cc:240] Negotiation successful
0619 18:44:55.471813 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1014,"thread_start_us":142,"threads_started":1}
I20260619 18:44:55.474330 2719 heartbeater.cc:347] Connected to a master server at 127.21.143.126:42297
I20260619 18:44:55.474601 2719 heartbeater.cc:464] Registering TS with master...
I20260619 18:44:55.475188 2719 heartbeater.cc:511] Master 127.21.143.126:42297 requested a full tablet report, sending...
W20260619 18:44:55.476864 2719 heartbeater.cc:644] Failed to heartbeat to 127.21.143.126:42297 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 18:44:55.476997 2719 heartbeater.cc:464] Registering TS with master...
I20260619 18:44:55.477183 2719 heartbeater.cc:511] Master 127.21.143.126:42297 requested a full tablet report, sending...
I20260619 18:44:55.479455 2719 heartbeater.cc:464] Registering TS with master...
I20260619 18:44:55.479626 2719 heartbeater.cc:511] Master 127.21.143.126:42297 requested a full tablet report, sending...
W20260619 18:44:55.480496 2719 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 18:44:55.641147 2722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:55.641520 2722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:55.641599 2722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:55.647702 2722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:55.647796 2722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:55.647890 2722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.67
I20260619 18:44:55.656528 2722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:42673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.143.84:33345/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.67:41433
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.143.67
--webserver_port=37187
--tserver_master_addrs=127.21.143.126:42297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.67
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:44:55.658123 2722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:55.659612 2722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:44:55.672927 2728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:55.672931 2730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:55.673507 2722 server_base.cc:1034] running on GCE node
W20260619 18:44:55.677122 2727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:55.677727 2722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:55.678606 2722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:55.681917 2722 hybrid_clock.cc:648] HybridClock initialized: now 1781894695681888 us; error 56 us; skew 500 ppm
Jun 19 18:44:55 dist-test-slave-mld5 krb5kdc[1390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894695, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:55.684911 2722 init.cc:377] Logged in from keytab as kudu/127.21.143.67@KRBTEST.COM (short username kudu)
I20260619 18:44:55.685880 2722 webserver.cc:460] Webserver started at http://127.21.143.67:37187/ using document root <none> and password file <none>
I20260619 18:44:55.686482 2722 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:55.686553 2722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 18:44:55 dist-test-slave-mld5 krb5kdc[1390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894695, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for HTTP/127.21.143.84@KRBTEST.COM
I20260619 18:44:55.714700 2722 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:44:55.721148 2745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:55.722955 2722 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260619 18:44:55.723119 2722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-2/wal
uuid: "9db7e67ae5ef406b9111606b55924c8f"
format_stamp: "Formatted at 2026-06-19 18:44:54 on dist-test-slave-mld5"
server_key: "cd00481e2bbbeba6cf1b00829dd7167d"
server_key_iv: "5332bedecc0c09036ce8cf618727d68c"
server_key_version: "kuduclusterkey@0"
I20260619 18:44:55.723912 2722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:55.754967 2722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:55.760319 2722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:55.760556 2722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:55.761307 2722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:44:55.762589 2722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:44:55.762656 2722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:55.762717 2722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:44:55.762749 2722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:55.774026 2722 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.67:41433
I20260619 18:44:55.774088 2858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.67:41433 every 8 connection(s)
I20260619 18:44:55.775411 2722 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
I20260619 18:44:55.779364 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 2722
Jun 19 18:44:55 dist-test-slave-mld5 krb5kdc[1390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894695, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:44:55.792713 2863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:55.780111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43348 (local address 127.21.143.126:42297)
0619 18:44:55.780470 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:55.780479 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:55.780759 (+ 280us) server_negotiation.cc:407] Connection header received
0619 18:44:55.780891 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:55.780894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:55.780939 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:55.781019 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:55.781986 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:55.782629 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:55.784365 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:55.784771 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:55.785021 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:55.787185 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:55.787188 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:55.787191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:55.787224 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:55.789425 (+ 2201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:55.790881 (+ 1456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:55.790885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:55.790887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:55.791019 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:55.791522 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:55.791525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:55.791527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:55.791809 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:55.792048 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:55.792265 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 18:44:55.792393 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":289,"spinlock_wait_cycles":1280,"thread_start_us":112,"threads_started":1}
I20260619 18:44:55.793522 2587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:55.777486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:42101 (local address 127.21.143.126:42297)
0619 18:44:55.777649 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:55.777658 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:55.778783 (+ 1125us) server_negotiation.cc:407] Connection header received
0619 18:44:55.779987 (+ 1204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:55.779992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:55.780056 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:55.780140 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:55.781957 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:55.782860 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:55.784341 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:55.784771 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:55.784931 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:55.788551 (+ 3620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:55.788556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:55.788560 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:55.788606 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:55.790207 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:55.790881 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:55.790885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:55.790887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:55.791005 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:55.791669 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:55.791672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:55.791673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:55.791853 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:55.791919 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:55.793217 (+ 1298us) server_negotiation.cc:299] Negotiation successful
0619 18:44:55.793348 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":104,"server-negotiator.queue_time_us":71,"spinlock_wait_cycles":6144}
I20260619 18:44:55.795145 2861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:55.778044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42297 (local address 127.21.143.67:42101)
0619 18:44:55.778621 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:55.778667 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:55.779700 (+ 1033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:55.780613 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:55.780623 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:55.781021 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:55.781791 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:55.781813 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:55.782996 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:55.783000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:55.784217 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:55.784226 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:55.785386 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:55.785391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:55.785627 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:55.786247 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:55.786267 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:55.788369 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:55.790402 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:55.790410 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:55.790432 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:55.790775 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:55.791355 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:55.791358 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:55.791360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:55.791557 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:55.792596 (+ 1039us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:55.792602 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:55.792940 (+ 338us) client_negotiation.cc:769] Sending connection context
0619 18:44:55.793845 (+ 905us) client_negotiation.cc:240] Negotiation successful
0619 18:44:55.794203 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":143,"threads_started":1}
I20260619 18:44:55.796226 2859 heartbeater.cc:347] Connected to a master server at 127.21.143.126:42297
I20260619 18:44:55.796468 2859 heartbeater.cc:464] Registering TS with master...
I20260619 18:44:55.796965 2859 heartbeater.cc:511] Master 127.21.143.126:42297 requested a full tablet report, sending...
I20260619 18:44:55.798318 2386 ts_manager.cc:194] Registered new tserver with Master: 9db7e67ae5ef406b9111606b55924c8f (127.21.143.67:41433)
I20260619 18:44:55.799804 2386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.67@KRBTEST.COM'} at 127.21.143.67:42101
Jun 19 18:44:56 dist-test-slave-mld5 krb5kdc[1390](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781894696, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:56.155984 2584 heartbeater.cc:464] Registering TS with master...
I20260619 18:44:56.156283 2584 heartbeater.cc:511] Master 127.21.143.126:42297 requested a full tablet report, sending...
I20260619 18:44:56.156975 2386 ts_manager.cc:194] Registered new tserver with Master: 0feef268011f4c35b898548bbd85d842 (127.21.143.65:37153)
I20260619 18:44:56.157581 2386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.65@KRBTEST.COM'} at 127.21.143.65:56713
Jun 19 18:44:56 dist-test-slave-mld5 krb5kdc[1390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894696, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for HTTP/127.21.143.84@KRBTEST.COM
I20260619 18:44:56.480727 2719 heartbeater.cc:464] Registering TS with master...
I20260619 18:44:56.481020 2719 heartbeater.cc:511] Master 127.21.143.126:42297 requested a full tablet report, sending...
I20260619 18:44:56.481742 2386 ts_manager.cc:194] Registered new tserver with Master: 17f2fb1d1a6a4f3ea9abaada1d9e2d59 (127.21.143.66:45009)
I20260619 18:44:56.482458 2386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.66@KRBTEST.COM'} at 127.21.143.66:57625
I20260619 18:44:56.492177 22077 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:44:56.505863 2873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:56.493879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52340 (local address 127.21.143.126:42297)
0619 18:44:56.494230 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:56.494237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:56.495723 (+ 1486us) server_negotiation.cc:407] Connection header received
0619 18:44:56.495773 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:56.495776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:56.495833 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:56.495931 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:56.496969 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:56.498073 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:56.498938 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:56.499282 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:56.499409 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:56.501196 (+ 1787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:56.501199 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:56.501202 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:56.501233 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:56.503370 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:56.503860 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:56.503863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:56.503865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:56.503965 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:56.504280 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:56.504283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:56.504285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:56.504520 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:56.504648 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:56.505542 (+ 894us) server_negotiation.cc:299] Negotiation successful
0619 18:44:56.505685 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":71,"threads_started":1}
I20260619 18:44:56.511291 2386 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52340:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:44:56.516602 2386 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:44:56.545744 2883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:56.538294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:41433 (local address 127.0.0.1:47262)
0619 18:44:56.540614 (+ 2320us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:56.540673 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:56.540802 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:56.541770 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:56.541777 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:56.541802 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:44:56.541956 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:56.541963 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:56.543320 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:56.543326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:56.544212 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:56.544222 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:56.545182 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:56.545186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:56.545309 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:56.545349 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 18:44:56.545467 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 18:44:56.545593 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2125,"thread_start_us":116,"threads_started":1}
I20260619 18:44:56.546523 2881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:56.539201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47262 (local address 127.21.143.67:41433)
0619 18:44:56.541317 (+ 2116us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:56.541325 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:56.541352 (+ 27us) server_negotiation.cc:407] Connection header received
0619 18:44:56.541438 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:56.541442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:56.541617 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:56.541794 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:44:56.542088 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:56.543168 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:56.544344 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:56.545054 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:56.545185 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:56.545357 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:56.545498 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 18:44:56.545593 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1972,"thread_start_us":136,"threads_started":1}
I20260619 18:44:56.549847 2793 tablet_service.cc:1467] Processing CreateTablet for tablet 4236d0923f9a4c3fa498702f88593ba2 (DEFAULT_TABLE table=test-table [id=70a2de514e76483da36fa0ef5b126c31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:44:56.551095 2793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4236d0923f9a4c3fa498702f88593ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:44:56.553092 2880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:56.540253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:45009 (local address 127.0.0.1:41414)
0619 18:44:56.547446 (+ 7193us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:56.547461 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:56.547548 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:56.548551 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:56.548555 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:56.548571 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:44:56.548663 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:56.548668 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:56.550104 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:56.550107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:56.551345 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:56.551356 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:56.552650 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:56.552654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:56.552753 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:56.552791 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 18:44:56.552873 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 18:44:56.552948 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7131,"thread_start_us":92,"threads_started":1}
I20260619 18:44:56.555011 2882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:56.539200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56646 (local address 127.21.143.65:37153)
0619 18:44:56.541076 (+ 1876us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:56.541083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:56.549758 (+ 8675us) server_negotiation.cc:407] Connection header received
0619 18:44:56.550003 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:56.550009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:56.550191 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:56.550304 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:44:56.550815 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:56.551940 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:56.553420 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:56.554219 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:56.554301 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:56.554431 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:56.554779 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 18:44:56.554877 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1775,"thread_start_us":172,"threads_started":1}
I20260619 18:44:56.555433 2883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:56.538353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:37153 (local address 127.0.0.1:56646)
0619 18:44:56.546108 (+ 7755us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:56.549661 (+ 3553us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:56.549839 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:56.550529 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:56.550532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:56.550548 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:44:56.550628 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:56.550633 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:56.552226 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:56.552229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:56.553276 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:56.553287 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:56.554524 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:56.554527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:56.554615 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:56.554636 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 18:44:56.555203 (+ 567us) client_negotiation.cc:240] Negotiation successful
0619 18:44:56.555283 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7701,"thread_start_us":84,"threads_started":1}
I20260619 18:44:56.555640 2884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:56.540424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41414 (local address 127.21.143.66:45009)
0619 18:44:56.547876 (+ 7452us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:56.547884 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:56.547911 (+ 27us) server_negotiation.cc:407] Connection header received
0619 18:44:56.547996 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:56.548002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:56.548232 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:56.548393 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:44:56.548795 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:56.549968 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:56.551494 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:56.552351 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:56.552452 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:56.552590 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:56.555384 (+ 2794us) server_negotiation.cc:299] Negotiation successful
0619 18:44:56.555490 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7314,"thread_start_us":62,"threads_started":1}
I20260619 18:44:56.558423 2644 tablet_service.cc:1467] Processing CreateTablet for tablet 4236d0923f9a4c3fa498702f88593ba2 (DEFAULT_TABLE table=test-table [id=70a2de514e76483da36fa0ef5b126c31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:44:56.558423 2517 tablet_service.cc:1467] Processing CreateTablet for tablet 4236d0923f9a4c3fa498702f88593ba2 (DEFAULT_TABLE table=test-table [id=70a2de514e76483da36fa0ef5b126c31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:44:56.559731 2644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4236d0923f9a4c3fa498702f88593ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:44:56.559753 2517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4236d0923f9a4c3fa498702f88593ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:44:56.574846 2886 tablet_bootstrap.cc:492] T 4236d0923f9a4c3fa498702f88593ba2 P 0feef268011f4c35b898548bbd85d842: Bootstrap starting.
I20260619 18:44:56.577510 2886 tablet_bootstrap.cc:654] T 4236d0923f9a4c3fa498702f88593ba2 P 0feef268011f4c35b898548bbd85d842: Neither blocks nor log segments found. Creating new log.
I20260619 18:44:56.578936 2886 log.cc:824] T 4236d0923f9a4c3fa498702f88593ba2 P 0feef268011f4c35b898548bbd85d842: Log is configured to *not* fsync() on all Append() calls
I20260619 18:44:56.579655 2885 tablet_bootstrap.cc:492] T 4236d0923f9a4c3fa498702f88593ba2 P 17f2fb1d1a6a4f3ea9abaada1d9e2d59: Bootstrap starting.
I20260619 18:44:56.581926 2885 tablet_bootstrap.cc:654] T 4236d0923f9a4c3fa498702f88593ba2 P 17f2fb1d1a6a4f3ea9abaada1d9e2d59: Neither blocks nor log segments found. Creating new log.
I20260619 18:44:56.583017 2885 log.cc:824] T 4236d0923f9a4c3fa498702f88593ba2 P 17f2fb1d1a6a4f3ea9abaada1d9e2d59: Log is configured to *not* fsync() on all Append() calls
I20260619 18:44:56.584124 2887 tablet_bootstrap.cc:492] T 4236d0923f9a4c3fa498702f88593ba2 P 9db7e67ae5ef406b9111606b55924c8f: Bootstrap starting.
I20260619 18:44:56.585289 2886 tablet_bootstrap.cc:492] T 4236d0923f9a4c3fa498702f88593ba2 P 0feef268011f4c35b898548bbd85d842: No bootstrap required, opened a new log
I20260619 18:44:56.585536 2886 ts_tablet_manager.cc:1397] T 4236d0923f9a4c3fa498702f88593ba2 P 0feef268011f4c35b898548bbd85d842: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.002s
I20260619 18:44:56.586289 2887 tablet_bootstrap.cc:654] T 4236d0923f9a4c3fa498702f88593ba2 P 9db7e67ae5ef406b9111606b55924c8f: Neither blocks nor log segments found. Creating new log.
I20260619 18:44:56.587083 2887 log.cc:824] T 4236d0923f9a4c3fa498702f88593ba2 P 9db7e67ae5ef406b9111606b55924c8f: Log is configured to *not* fsync() on all Append() calls
I20260619 18:44:56.589166 2887 tablet_bootstrap.cc:492] T 4236d0923f9a4c3fa498702f88593ba2 P 9db7e67ae5ef406b9111606b55924c8f: No bootstrap required, opened a new log
I20260619 18:44:56.589370 2887 ts_tablet_manager.cc:1397] T 4236d0923f9a4c3fa498702f88593ba2 P 9db7e67ae5ef406b9111606b55924c8f: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 18:44:56.590107 2886 raft_consensus.cc:348] T 4236d0923f9a4c3fa498702f88593ba2 P 0feef268011f4c35b898548bbd85d842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db7e67ae5ef406b9111606b55924c8f" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 41433 } } peers { permanent_uuid: "0feef268011f4c35b898548bbd85d842" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 37153 } } peers { permanent_uuid: "17f2fb1d1a6a4f3ea9abaada1d9e2d59" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 45009 } }
I20260619 18:44:56.590533 2886 raft_consensus.cc:374] T 4236d0923f9a4c3fa498702f88593ba2 P 0feef268011f4c35b898548bbd85d842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:44:56.590663 2886 raft_consensus.cc:729] T 4236d0923f9a4c3fa498702f88593ba2 P 0feef268011f4c35b898548bbd85d842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0feef268011f4c35b898548bbd85d842, State: Initialized, Role: FOLLOWER
I20260619 18:44:56.591277 2886 consensus_queue.cc:260] T 4236d0923f9a4c3fa498702f88593ba2 P 0feef268011f4c35b898548bbd85d842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db7e67ae5ef406b9111606b55924c8f" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 41433 } } peers { permanent_uuid: "0feef268011f4c35b898548bbd85d842" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 37153 } } peers { permanent_uuid: "17f2fb1d1a6a4f3ea9abaada1d9e2d59" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 45009 } }
I20260619 18:44:56.593699 2886 ts_tablet_manager.cc:1428] T 4236d0923f9a4c3fa498702f88593ba2 P 0feef268011f4c35b898548bbd85d842: Time spent starting tablet: real 0.008s user 0.002s sys 0.004s
I20260619 18:44:56.593891 2887 raft_consensus.cc:348] T 4236d0923f9a4c3fa498702f88593ba2 P 9db7e67ae5ef406b9111606b55924c8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db7e67ae5ef406b9111606b55924c8f" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 41433 } } peers { permanent_uuid: "0feef268011f4c35b898548bbd85d842" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 37153 } } peers { permanent_uuid: "17f2fb1d1a6a4f3ea9abaada1d9e2d59" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 45009 } }
I20260619 18:44:56.594130 2584 heartbeater.cc:503] Master 127.21.143.126:42297 was elected leader, sending a full tablet report...
I20260619 18:44:56.594153 2887 raft_consensus.cc:374] T 4236d0923f9a4c3fa498702f88593ba2 P 9db7e67ae5ef406b9111606b55924c8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:44:56.594199 2887 raft_consensus.cc:729] T 4236d0923f9a4c3fa498702f88593ba2 P 9db7e67ae5ef406b9111606b55924c8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db7e67ae5ef406b9111606b55924c8f, State: Initialized, Role: FOLLOWER
I20260619 18:44:56.596482 2885 tablet_bootstrap.cc:492] T 4236d0923f9a4c3fa498702f88593ba2 P 17f2fb1d1a6a4f3ea9abaada1d9e2d59: No bootstrap required, opened a new log
I20260619 18:44:56.596715 2885 ts_tablet_manager.cc:1397] T 4236d0923f9a4c3fa498702f88593ba2 P 17f2fb1d1a6a4f3ea9abaada1d9e2d59: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20260619 18:44:56.599690 2887 consensus_queue.cc:260] T 4236d0923f9a4c3fa498702f88593ba2 P 9db7e67ae5ef406b9111606b55924c8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db7e67ae5ef406b9111606b55924c8f" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 41433 } } peers { permanent_uuid: "0feef268011f4c35b898548bbd85d842" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 37153 } } peers { permanent_uuid: "17f2fb1d1a6a4f3ea9abaada1d9e2d59" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 45009 } }
I20260619 18:44:56.600520 2885 raft_consensus.cc:348] T 4236d0923f9a4c3fa498702f88593ba2 P 17f2fb1d1a6a4f3ea9abaada1d9e2d59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db7e67ae5ef406b9111606b55924c8f" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 41433 } } peers { permanent_uuid: "0feef268011f4c35b898548bbd85d842" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 37153 } } peers { permanent_uuid: "17f2fb1d1a6a4f3ea9abaada1d9e2d59" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 45009 } }
I20260619 18:44:56.600809 2885 raft_consensus.cc:374] T 4236d0923f9a4c3fa498702f88593ba2 P 17f2fb1d1a6a4f3ea9abaada1d9e2d59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:44:56.600888 2885 raft_consensus.cc:729] T 4236d0923f9a4c3fa498702f88593ba2 P 17f2fb1d1a6a4f3ea9abaada1d9e2d59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17f2fb1d1a6a4f3ea9abaada1d9e2d59, State: Initialized, Role: FOLLOWER
I20260619 18:44:56.601176 2887 ts_tablet_manager.cc:1428] T 4236d0923f9a4c3fa498702f88593ba2 P 9db7e67ae5ef406b9111606b55924c8f: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260619 18:44:56.601415 2885 consensus_queue.cc:260] T 4236d0923f9a4c3fa498702f88593ba2 P 17f2fb1d1a6a4f3ea9abaada1d9e2d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db7e67ae5ef406b9111606b55924c8f" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 41433 } } peers { permanent_uuid: "0feef268011f4c35b898548bbd85d842" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 37153 } } peers { permanent_uuid: "17f2fb1d1a6a4f3ea9abaada1d9e2d59" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 45009 } }
I20260619 18:44:56.601625 2859 heartbeater.cc:503] Master 127.21.143.126:42297 was elected leader, sending a full tablet report...
I20260619 18:44:56.602491 2885 ts_tablet_manager.cc:1428] T 4236d0923f9a4c3fa498702f88593ba2 P 17f2fb1d1a6a4f3ea9abaada1d9e2d59: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 18:44:56.602826 2719 heartbeater.cc:503] Master 127.21.143.126:42297 was elected leader, sending a full tablet report...
W20260619 18:44:56.623286 2585 tablet.cc:2369] T 4236d0923f9a4c3fa498702f88593ba2 P 0feef268011f4c35b898548bbd85d842: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:44:56.706681 2720 tablet.cc:2369] T 4236d0923f9a4c3fa498702f88593ba2 P 17f2fb1d1a6a4f3ea9abaada1d9e2d59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:44:56.729902 2893 raft_consensus.cc:482] T 4236d0923f9a4c3fa498702f88593ba2 P 17f2fb1d1a6a4f3ea9abaada1d9e2d59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:44:56.730141 2893 raft_consensus.cc:504] T 4236d0923f9a4c3fa498702f88593ba2 P 17f2fb1d1a6a4f3ea9abaada1d9e2d59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db7e67ae5ef406b9111606b55924c8f" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 41433 } } peers { permanent_uuid: "0feef268011f4c35b898548bbd85d842" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 37153 } } peers { permanent_uuid: "17f2fb1d1a6a4f3ea9abaada1d9e2d59" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 45009 } }
I20260619 18:44:56.731401 2893 leader_election.cc:288] T 4236d0923f9a4c3fa498702f88593ba2 P 17f2fb1d1a6a4f3ea9abaada1d9e2d59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9db7e67ae5ef406b9111606b55924c8f (127.21.143.67:41433), 0feef268011f4c35b898548bbd85d842 (127.21.143.65:37153)
I20260619 18:44:56.739482 2882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:56.732226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:52149 (local address 127.21.143.65:37153)
0619 18:44:56.732364 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:56.732369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:56.734147 (+ 1778us) server_negotiation.cc:407] Connection header received
0619 18:44:56.734350 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:56.734353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:56.734401 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:56.734448 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:44:56.734923 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:56.735696 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:56.737966 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:56.738550 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:56.738650 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:56.738717 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:56.739228 (+ 511us) server_negotiation.cc:299] Negotiation successful
0619 18:44:56.739344 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 18:44:56.740154 2894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:56.731828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:37153 (local address 127.21.143.66:52149)
0619 18:44:56.734062 (+ 2234us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:56.734077 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:56.734225 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:56.734617 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:56.734620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:56.734637 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:44:56.734794 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:56.734800 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:56.735849 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:56.735853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:56.737803 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:56.737818 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:56.738818 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:56.738822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:56.739060 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:56.739086 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 18:44:56.739909 (+ 823us) client_negotiation.cc:240] Negotiation successful
0619 18:44:56.739985 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2155,"mutex_wait_us":118,"thread_start_us":108,"threads_started":1}
I20260619 18:44:56.740986 2520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4236d0923f9a4c3fa498702f88593ba2" candidate_uuid: "17f2fb1d1a6a4f3ea9abaada1d9e2d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0feef268011f4c35b898548bbd85d842" is_pre_election: true
I20260619 18:44:56.741343 2520 raft_consensus.cc:2454] T 4236d0923f9a4c3fa498702f88593ba2 P 0feef268011f4c35b898548bbd85d842 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17f2fb1d1a6a4f3ea9abaada1d9e2d59 in term 0.
I20260619 18:44:56.741921 2607 leader_election.cc:302] T 4236d0923f9a4c3fa498702f88593ba2 P 17f2fb1d1a6a4f3ea9abaada1d9e2d59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0feef268011f4c35b898548bbd85d842, 17f2fb1d1a6a4f3ea9abaada1d9e2d59; no voters:
I20260619 18:44:56.742245 2881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:56.732872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:59001 (local address 127.21.143.67:41433)
0619 18:44:56.732988 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:56.732992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:56.736205 (+ 3213us) server_negotiation.cc:407] Connection header received
0619 18:44:56.736432 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:56.736436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:56.736483 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:56.736526 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:44:56.736863 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:56.737559 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:56.740837 (+ 3278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:56.741411 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:56.741508 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:56.741554 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:56.742044 (+ 490us) server_negotiation.cc:299] Negotiation successful
0619 18:44:56.742129 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 18:44:56.742250 2893 raft_consensus.cc:2781] T 4236d0923f9a4c3fa498702f88593ba2 P 17f2fb1d1a6a4f3ea9abaada1d9e2d59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:44:56.742338 2893 raft_consensus.cc:482] T 4236d0923f9a4c3fa498702f88593ba2 P 17f2fb1d1a6a4f3ea9abaada1d9e2d59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:44:56.742388 2893 raft_consensus.cc:3037] T 4236d0923f9a4c3fa498702f88593ba2 P 17f2fb1d1a6a4f3ea9abaada1d9e2d59 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:44:56.742609 2895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:56.732512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:41433 (local address 127.21.143.66:59001)
0619 18:44:56.736119 (+ 3607us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:56.736132 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:56.736315 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:56.736639 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:56.736642 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:56.736654 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:44:56.736748 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:56.736753 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:56.737723 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:56.737725 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:56.738954 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:56.738965 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:56.741789 (+ 2824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:56.741793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:56.741896 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:56.741919 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 18:44:56.742382 (+ 463us) client_negotiation.cc:240] Negotiation successful
0619 18:44:56.742454 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3547,"mutex_wait_us":1,"thread_start_us":96,"threads_started":1}
I20260619 18:44:56.743388 2813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4236d0923f9a4c3fa498702f88593ba2" candidate_uuid: "17f2fb1d1a6a4f3ea9abaada1d9e2d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9db7e67ae5ef406b9111606b55924c8f" is_pre_election: true
I20260619 18:44:56.743690 2813 raft_consensus.cc:2454] T 4236d0923f9a4c3fa498702f88593ba2 P 9db7e67ae5ef406b9111606b55924c8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17f2fb1d1a6a4f3ea9abaada1d9e2d59 in term 0.
I20260619 18:44:56.743801 2893 raft_consensus.cc:504] T 4236d0923f9a4c3fa498702f88593ba2 P 17f2fb1d1a6a4f3ea9abaada1d9e2d59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db7e67ae5ef406b9111606b55924c8f" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 41433 } } peers { permanent_uuid: "0feef268011f4c35b898548bbd85d842" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 37153 } } peers { permanent_uuid: "17f2fb1d1a6a4f3ea9abaada1d9e2d59" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 45009 } }
I20260619 18:44:56.744196 2893 leader_election.cc:288] T 4236d0923f9a4c3fa498702f88593ba2 P 17f2fb1d1a6a4f3ea9abaada1d9e2d59 [CANDIDATE]: Term 1 election: Requested vote from peers 9db7e67ae5ef406b9111606b55924c8f (127.21.143.67:41433), 0feef268011f4c35b898548bbd85d842 (127.21.143.65:37153)
I20260619 18:44:56.745186 2520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4236d0923f9a4c3fa498702f88593ba2" candidate_uuid: "17f2fb1d1a6a4f3ea9abaada1d9e2d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0feef268011f4c35b898548bbd85d842"
I20260619 18:44:56.745323 2520 raft_consensus.cc:3037] T 4236d0923f9a4c3fa498702f88593ba2 P 0feef268011f4c35b898548bbd85d842 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:44:56.746675 2813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4236d0923f9a4c3fa498702f88593ba2" candidate_uuid: "17f2fb1d1a6a4f3ea9abaada1d9e2d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9db7e67ae5ef406b9111606b55924c8f"
I20260619 18:44:56.746661 2520 raft_consensus.cc:2454] T 4236d0923f9a4c3fa498702f88593ba2 P 0feef268011f4c35b898548bbd85d842 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17f2fb1d1a6a4f3ea9abaada1d9e2d59 in term 1.
I20260619 18:44:56.746829 2813 raft_consensus.cc:3037] T 4236d0923f9a4c3fa498702f88593ba2 P 9db7e67ae5ef406b9111606b55924c8f [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:44:56.747128 2607 leader_election.cc:302] T 4236d0923f9a4c3fa498702f88593ba2 P 17f2fb1d1a6a4f3ea9abaada1d9e2d59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0feef268011f4c35b898548bbd85d842, 17f2fb1d1a6a4f3ea9abaada1d9e2d59; no voters:
I20260619 18:44:56.747334 2893 raft_consensus.cc:2781] T 4236d0923f9a4c3fa498702f88593ba2 P 17f2fb1d1a6a4f3ea9abaada1d9e2d59 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:44:56.747536 2893 raft_consensus.cc:686] T 4236d0923f9a4c3fa498702f88593ba2 P 17f2fb1d1a6a4f3ea9abaada1d9e2d59 [term 1 LEADER]: Becoming Leader. State: Replica: 17f2fb1d1a6a4f3ea9abaada1d9e2d59, State: Running, Role: LEADER
I20260619 18:44:56.747880 2893 consensus_queue.cc:237] T 4236d0923f9a4c3fa498702f88593ba2 P 17f2fb1d1a6a4f3ea9abaada1d9e2d59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db7e67ae5ef406b9111606b55924c8f" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 41433 } } peers { permanent_uuid: "0feef268011f4c35b898548bbd85d842" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 37153 } } peers { permanent_uuid: "17f2fb1d1a6a4f3ea9abaada1d9e2d59" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 45009 } }
I20260619 18:44:56.748171 2813 raft_consensus.cc:2454] T 4236d0923f9a4c3fa498702f88593ba2 P 9db7e67ae5ef406b9111606b55924c8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17f2fb1d1a6a4f3ea9abaada1d9e2d59 in term 1.
I20260619 18:44:56.751554 2386 catalog_manager.cc:5495] T 4236d0923f9a4c3fa498702f88593ba2 P 17f2fb1d1a6a4f3ea9abaada1d9e2d59 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17f2fb1d1a6a4f3ea9abaada1d9e2d59 (127.21.143.66). New cstate: current_term: 1 leader_uuid: "17f2fb1d1a6a4f3ea9abaada1d9e2d59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db7e67ae5ef406b9111606b55924c8f" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 41433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0feef268011f4c35b898548bbd85d842" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 37153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17f2fb1d1a6a4f3ea9abaada1d9e2d59" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 45009 } health_report { overall_health: HEALTHY } } }
Jun 19 18:44:56 dist-test-slave-mld5 krb5kdc[1390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894696, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for HTTP/127.21.143.84@KRBTEST.COM
W20260619 18:44:56.777205 2860 tablet.cc:2369] T 4236d0923f9a4c3fa498702f88593ba2 P 9db7e67ae5ef406b9111606b55924c8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:44:56.797232 2884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:56.791479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41420 (local address 127.21.143.66:45009)
0619 18:44:56.791605 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:56.791610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:56.791942 (+ 332us) server_negotiation.cc:407] Connection header received
0619 18:44:56.792041 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:56.792044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:56.792117 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:56.792243 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:44:56.792679 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:56.793485 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:56.794871 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:56.795150 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:56.795265 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:56.795808 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:44:56.796433 (+ 625us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:44:56.796555 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:56.796984 (+ 429us) server_negotiation.cc:299] Negotiation successful
0619 18:44:56.797051 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 18:44:56.804742 2520 raft_consensus.cc:1261] T 4236d0923f9a4c3fa498702f88593ba2 P 0feef268011f4c35b898548bbd85d842 [term 1 FOLLOWER]: Refusing update from remote peer 17f2fb1d1a6a4f3ea9abaada1d9e2d59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:44:56.805001 2813 raft_consensus.cc:1261] T 4236d0923f9a4c3fa498702f88593ba2 P 9db7e67ae5ef406b9111606b55924c8f [term 1 FOLLOWER]: Refusing update from remote peer 17f2fb1d1a6a4f3ea9abaada1d9e2d59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:44:56.805569 2893 consensus_queue.cc:1035] T 4236d0923f9a4c3fa498702f88593ba2 P 17f2fb1d1a6a4f3ea9abaada1d9e2d59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0feef268011f4c35b898548bbd85d842" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 37153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:44:56.805757 2897 consensus_queue.cc:1035] T 4236d0923f9a4c3fa498702f88593ba2 P 17f2fb1d1a6a4f3ea9abaada1d9e2d59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9db7e67ae5ef406b9111606b55924c8f" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 41433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:44:56.828814 2902 mvcc.cc:205] Tried to move back new op lower bound from 7298640678103158784 to 7298640677882880000. Current Snapshot: MvccSnapshot[applied={T|T < 7298640678103158784}]
I20260619 18:44:56.829015 2904 mvcc.cc:205] Tried to move back new op lower bound from 7298640678103158784 to 7298640677882880000. Current Snapshot: MvccSnapshot[applied={T|T < 7298640678103158784}]
I20260619 18:44:56.841495 2385 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:44:56.841727 2385 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:44:56.844970 2385 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 21de688ffc8049088c10cb7d07da7210: Sending DeleteTablet for 3 replicas of tablet 4236d0923f9a4c3fa498702f88593ba2
I20260619 18:44:56.845914 2517 tablet_service.cc:1514] Processing DeleteTablet for tablet 4236d0923f9a4c3fa498702f88593ba2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:44:56 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:56646
I20260619 18:44:56.846536 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 2424
I20260619 18:44:56.849062 2644 tablet_service.cc:1514] Processing DeleteTablet for tablet 4236d0923f9a4c3fa498702f88593ba2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:44:56 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:41414
I20260619 18:44:56.849354 2793 tablet_service.cc:1514] Processing DeleteTablet for tablet 4236d0923f9a4c3fa498702f88593ba2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:44:56 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:47262
I20260619 18:44:56.850263 2912 tablet_replica.cc:330] T 4236d0923f9a4c3fa498702f88593ba2 P 17f2fb1d1a6a4f3ea9abaada1d9e2d59: stopping tablet replica
I20260619 18:44:56.850621 2912 raft_consensus.cc:2229] T 4236d0923f9a4c3fa498702f88593ba2 P 17f2fb1d1a6a4f3ea9abaada1d9e2d59 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:44:56.851086 2912 raft_consensus.cc:2258] T 4236d0923f9a4c3fa498702f88593ba2 P 17f2fb1d1a6a4f3ea9abaada1d9e2d59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:44:56.852833 2912 ts_tablet_manager.cc:1905] T 4236d0923f9a4c3fa498702f88593ba2 P 17f2fb1d1a6a4f3ea9abaada1d9e2d59: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 18:44:56.856595 2371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.143.65:37153 (error 108)
I20260619 18:44:56.856665 2912 ts_tablet_manager.cc:1918] T 4236d0923f9a4c3fa498702f88593ba2 P 17f2fb1d1a6a4f3ea9abaada1d9e2d59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:44:56.856751 2912 log.cc:1196] T 4236d0923f9a4c3fa498702f88593ba2 P 17f2fb1d1a6a4f3ea9abaada1d9e2d59: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-1/wal/wals/4236d0923f9a4c3fa498702f88593ba2
I20260619 18:44:56.857084 2912 ts_tablet_manager.cc:1939] T 4236d0923f9a4c3fa498702f88593ba2 P 17f2fb1d1a6a4f3ea9abaada1d9e2d59: Deleting consensus metadata
I20260619 18:44:56.858475 2373 catalog_manager.cc:4841] TS 17f2fb1d1a6a4f3ea9abaada1d9e2d59 (127.21.143.66:45009): tablet 4236d0923f9a4c3fa498702f88593ba2 (table test-table [id=70a2de514e76483da36fa0ef5b126c31]) successfully deleted
I20260619 18:44:56.867537 2913 tablet_replica.cc:330] T 4236d0923f9a4c3fa498702f88593ba2 P 9db7e67ae5ef406b9111606b55924c8f: stopping tablet replica
I20260619 18:44:56.867946 2913 raft_consensus.cc:2229] T 4236d0923f9a4c3fa498702f88593ba2 P 9db7e67ae5ef406b9111606b55924c8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:44:56.868331 2913 raft_consensus.cc:2258] T 4236d0923f9a4c3fa498702f88593ba2 P 9db7e67ae5ef406b9111606b55924c8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:44:56.869984 2913 ts_tablet_manager.cc:1905] T 4236d0923f9a4c3fa498702f88593ba2 P 9db7e67ae5ef406b9111606b55924c8f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 18:44:56.871739 2883 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:44:56.859209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:37153 (local address 127.0.0.1:56652)
0619 18:44:56.859414 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:56.859430 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:56.859522 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:56.871600 (+ 12078us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.65:37153: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":127}
W20260619 18:44:56.871951 2371 catalog_manager.cc:4584] TS 0feef268011f4c35b898548bbd85d842 (127.21.143.65:37153): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4236d0923f9a4c3fa498702f88593ba2: Network error: Client connection negotiation failed: client connection to 127.21.143.65:37153: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 18:44:56.872910 2913 ts_tablet_manager.cc:1918] T 4236d0923f9a4c3fa498702f88593ba2 P 9db7e67ae5ef406b9111606b55924c8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:44:56.873024 2913 log.cc:1196] T 4236d0923f9a4c3fa498702f88593ba2 P 9db7e67ae5ef406b9111606b55924c8f: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894542572984-22077-0/minicluster-data/ts-2/wal/wals/4236d0923f9a4c3fa498702f88593ba2
I20260619 18:44:56.873188 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 2588
I20260619 18:44:56.873349 2913 ts_tablet_manager.cc:1939] T 4236d0923f9a4c3fa498702f88593ba2 P 9db7e67ae5ef406b9111606b55924c8f: Deleting consensus metadata
I20260619 18:44:56.874868 2373 catalog_manager.cc:4841] TS 9db7e67ae5ef406b9111606b55924c8f (127.21.143.67:41433): tablet 4236d0923f9a4c3fa498702f88593ba2 (table test-table [id=70a2de514e76483da36fa0ef5b126c31]) successfully deleted
I20260619 18:44:56.884495 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 2722
I20260619 18:44:56.893440 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 2353
I20260619 18:44:56.901288 22077 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 18:44:57.246129 22077 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 18:44:57.246249 22077 mini_ranger.cc:66] Stopping Ranger...
I20260619 18:44:57.591558 22077 mini_ranger.cc:68] Stopped Ranger
2026-06-19 18:44:57.591 UTC [1420] LOG: received smart shutdown request
2026-06-19 18:44:57.595 UTC [1420] LOG: background worker "logical replication launcher" (PID 1429) exited with exit code 1
2026-06-19 18:44:57.595 UTC [1424] LOG: shutting down
2026-06-19 18:44:57.747 UTC [1420] LOG: database system is shut down
2026-06-19T18:44:57Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46743 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:44:57 dist-test-slave-mld5 krb5kdc[2923](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:44:57 dist-test-slave-mld5 krb5kdc[2923](info): set up 2 sockets
Jun 19 18:44:57 dist-test-slave-mld5 krb5kdc[2923](info): commencing operation
krb5kdc: starting...
W20260619 18:44:58.903484 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:44:58 dist-test-slave-mld5 krb5kdc[2923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:44:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:44:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.143.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
I20260619 18:44:59.019989 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:33747
--webserver_interface=127.21.143.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:36191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:33747,127.21.143.125:37115,127.21.143.124:43751
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:44:59.131798 2936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:59.132122 2936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:59.132192 2936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:59.136237 2936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:44:59.136327 2936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:59.136364 2936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:59.136430 2936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:44:59.136480 2936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:44:59.141908 2936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:36191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:33747,127.21.143.125:37115,127.21.143.124:43751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:33747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:44:59.143044 2936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:59.144097 2936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 18:44:59.152763 2936 server_base.cc:1034] running on GCE node
W20260619 18:44:59.152712 2941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:59.152712 2944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:59.152712 2942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:59.153584 2936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:59.154696 2936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:59.155886 2936 hybrid_clock.cc:648] HybridClock initialized: now 1781894699155864 us; error 36 us; skew 500 ppm
Jun 19 18:44:59 dist-test-slave-mld5 krb5kdc[2923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894699, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:59.158861 2936 init.cc:377] Logged in from keytab as kudu/127.21.143.126@KRBTEST.COM (short username kudu)
I20260619 18:44:59.159915 2936 webserver.cc:460] Webserver started at http://127.21.143.126:40355/ using document root <none> and password file <none>
I20260619 18:44:59.160547 2936 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:59.160616 2936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:44:59.160851 2936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:44:59.163026 2936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "4148ad498ae04b82a8d4a935fad20f3d"
format_stamp: "Formatted at 2026-06-19 18:44:59 on dist-test-slave-mld5"
server_key: "3908940dc38c48fff312e1dd734d420e"
server_key_iv: "33d63b9deadae551c065b55193d09593"
server_key_version: "encryptionkey@0"
I20260619 18:44:59.163708 2936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "4148ad498ae04b82a8d4a935fad20f3d"
format_stamp: "Formatted at 2026-06-19 18:44:59 on dist-test-slave-mld5"
server_key: "3908940dc38c48fff312e1dd734d420e"
server_key_iv: "33d63b9deadae551c065b55193d09593"
server_key_version: "encryptionkey@0"
I20260619 18:44:59.168058 2936 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:44:59.170782 2951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:59.172377 2936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:44:59.172559 2936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "4148ad498ae04b82a8d4a935fad20f3d"
format_stamp: "Formatted at 2026-06-19 18:44:59 on dist-test-slave-mld5"
server_key: "3908940dc38c48fff312e1dd734d420e"
server_key_iv: "33d63b9deadae551c065b55193d09593"
server_key_version: "encryptionkey@0"
I20260619 18:44:59.172694 2936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:59.226671 2936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:59.229744 2936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:59.229979 2936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:59.237030 2936 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:33747
I20260619 18:44:59.237337 3003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:33747 every 8 connection(s)
I20260619 18:44:59.238121 2936 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:44:59.241170 3004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:44:59.245224 3004 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33747 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:44:59.246539 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 2936
I20260619 18:44:59.246658 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:44:59.246932 22077 external_mini_cluster.cc:1428] Setting key 1322be27e9a662d5d938cbf759676824
WARNING: no policy specified for kudu/127.21.143.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.125@KRBTEST.COM" created.
Jun 19 18:44:59 dist-test-slave-mld5 krb5kdc[2923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894699, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:44:59.263480 3006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.247804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33747 (local address 127.0.0.1:39024)
0619 18:44:59.248408 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:59.248470 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:59.249303 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:59.249978 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:59.249984 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:59.250421 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:59.250655 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:59.250668 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.252552 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.252554 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:59.253278 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:59.253287 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.254177 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.254195 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:59.254286 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.254914 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:59.254930 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:59.258078 (+ 3148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:59.260472 (+ 2394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:59.260484 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:59.260498 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:59.260820 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:59.261149 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:59.261155 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:59.261160 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:59.261321 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:59.261985 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:59.261991 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:59.262153 (+ 162us) client_negotiation.cc:769] Sending connection context
0619 18:44:59.262304 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 18:44:59.262583 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":3,"thread_start_us":155,"threads_started":1}
I20260619 18:44:59.263492 3007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.247804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39024 (local address 127.21.143.126:33747)
0619 18:44:59.248374 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:59.248391 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:59.248595 (+ 204us) server_negotiation.cc:407] Connection header received
0619 18:44:59.249607 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:59.249612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:59.249787 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:59.249978 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:59.250866 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.252425 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.253399 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.254014 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.254230 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.258236 (+ 4006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:59.258241 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:59.258252 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:59.258271 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:59.260348 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.260937 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.260941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.260943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.261044 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.261438 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.261441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.261443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.261846 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:59.262009 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:59.262428 (+ 419us) server_negotiation.cc:299] Negotiation successful
0619 18:44:59.262694 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":448,"spinlock_wait_cycles":55040,"thread_start_us":164,"threads_started":1}
I20260619 18:44:59.266237 3004 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.143.125" port: 37115 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:44:59.267040 3006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.266771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.125:37115 (local address 127.0.0.1:32818)
0619 18:44:59.266888 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:59.266976 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.125:37115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 18:44:59.267349 2955 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.143.125:37115: connect: Connection refused (error 111)
I20260619 18:44:59.268239 3006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.268040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.125:37115 (local address 127.0.0.1:32834)
0619 18:44:59.268141 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:59.268185 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.125:37115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 18:44:59.268462 3004 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.143.125:37115: Network error: Client connection negotiation failed: client connection to 127.21.143.125:37115: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.125.keytab.
Entry for principal kudu/127.21.143.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.125.keytab.
I20260619 18:44:59.272147 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.125:37115
--webserver_interface=127.21.143.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.125.keytab
--principal=kudu/127.21.143.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:36191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:33747,127.21.143.125:37115,127.21.143.124:43751
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:44:59.317720 3004 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.143.125" port: 37115 } attempt: 1
I20260619 18:44:59.318634 3006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.318338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.125:37115 (local address 127.0.0.1:32840)
0619 18:44:59.318494 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:59.318558 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.125:37115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 18:44:59.319425 3006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.319176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.125:37115 (local address 127.0.0.1:32846)
0619 18:44:59.319271 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:59.319369 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.125:37115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 18:44:59.319581 3004 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.143.125:37115: Network error: Client connection negotiation failed: client connection to 127.21.143.125:37115: connect: Connection refused (error 111)
W20260619 18:44:59.384167 3011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:59.384435 3011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:59.384475 3011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 18:44:59.387749 3004 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.143.125" port: 37115 } attempt: 2
I20260619 18:44:59.388602 3006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.388288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.125:37115 (local address 127.0.0.1:32862)
0619 18:44:59.388460 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:59.388528 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.125:37115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260619 18:44:59.388635 3011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:44:59.388679 3011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:59.388728 3011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:59.388792 3011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:44:59.388828 3011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:44:59.389356 3006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.389160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.125:37115 (local address 127.0.0.1:32878)
0619 18:44:59.389260 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:59.389304 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.125:37115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 18:44:59.389493 3004 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.143.125:37115: Network error: Client connection negotiation failed: client connection to 127.21.143.125:37115: connect: Connection refused (error 111)
I20260619 18:44:59.394248 3011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:36191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:33747,127.21.143.125:37115,127.21.143.124:43751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.125.keytab
--principal=kudu/127.21.143.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.125:37115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.143.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:44:59.395285 3011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:59.396370 3011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 18:44:59.404654 3011 server_base.cc:1034] running on GCE node
W20260619 18:44:59.404603 3017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:59.404614 3016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:59.404588 3019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:59.405443 3011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:59.406544 3011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:59.407728 3011 hybrid_clock.cc:648] HybridClock initialized: now 1781894699407713 us; error 28 us; skew 500 ppm
Jun 19 18:44:59 dist-test-slave-mld5 krb5kdc[2923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894699, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:59.410362 3011 init.cc:377] Logged in from keytab as kudu/127.21.143.125@KRBTEST.COM (short username kudu)
I20260619 18:44:59.411206 3011 webserver.cc:460] Webserver started at http://127.21.143.125:34667/ using document root <none> and password file <none>
I20260619 18:44:59.411774 3011 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:59.411885 3011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:44:59.412115 3011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:44:59.414065 3011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-1/data/instance:
uuid: "702d62e07157426ba4805408701b1863"
format_stamp: "Formatted at 2026-06-19 18:44:59 on dist-test-slave-mld5"
server_key: "729b1ab6309eee69a6fb56013c11eff3"
server_key_iv: "b7b676328801f8426c80c3d588fc96d9"
server_key_version: "encryptionkey@0"
I20260619 18:44:59.414628 3011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-1/wal/instance:
uuid: "702d62e07157426ba4805408701b1863"
format_stamp: "Formatted at 2026-06-19 18:44:59 on dist-test-slave-mld5"
server_key: "729b1ab6309eee69a6fb56013c11eff3"
server_key_iv: "b7b676328801f8426c80c3d588fc96d9"
server_key_version: "encryptionkey@0"
I20260619 18:44:59.418228 3011 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 18:44:59.420547 3026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:59.421721 3011 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:44:59.421875 3011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-1/wal
uuid: "702d62e07157426ba4805408701b1863"
format_stamp: "Formatted at 2026-06-19 18:44:59 on dist-test-slave-mld5"
server_key: "729b1ab6309eee69a6fb56013c11eff3"
server_key_iv: "b7b676328801f8426c80c3d588fc96d9"
server_key_version: "encryptionkey@0"
I20260619 18:44:59.421983 3011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:59.444803 3011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:59.447842 3011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:59.448038 3011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:59.454962 3011 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.125:37115
I20260619 18:44:59.455241 3078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.125:37115 every 8 connection(s)
I20260619 18:44:59.456008 3011 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-1/data/info.pb
I20260619 18:44:59.458045 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 3011
I20260619 18:44:59.458138 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-1/wal/instance
I20260619 18:44:59.458418 22077 external_mini_cluster.cc:1428] Setting key 58b1309c1ab4c4438cd17c2b163bc5d9
I20260619 18:44:59.458810 3079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:44:59.463151 3079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33747 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.143.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.124@KRBTEST.COM" created.
Jun 19 18:44:59 dist-test-slave-mld5 krb5kdc[2923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894699, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.125@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:44:59.477934 3007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.465146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39034 (local address 127.21.143.126:33747)
0619 18:44:59.465283 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:59.465287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:59.466321 (+ 1034us) server_negotiation.cc:407] Connection header received
0619 18:44:59.467223 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:59.467227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:59.467295 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:59.467431 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:59.468594 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.469396 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.470554 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.470984 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.471114 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.473889 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:59.473897 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:59.473902 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:59.473937 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:59.475638 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.476231 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.476235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.476237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.476332 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.476688 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.476693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.476697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.476981 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:59.477087 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:59.477686 (+ 599us) server_negotiation.cc:299] Negotiation successful
0619 18:44:59.477810 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 18:44:59.478820 3082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.465639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33747 (local address 127.0.0.1:39034)
0619 18:44:59.466156 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:59.466197 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:59.467012 (+ 815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:59.467627 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:59.467634 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:59.467970 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:59.468444 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:59.468457 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.469540 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.469543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:59.470423 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:59.470433 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.471144 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.471147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:59.471375 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.471940 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:59.471965 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:59.473703 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:59.475755 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:59.475760 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:59.475772 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:59.476043 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:59.476435 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:59.476438 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:59.476439 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:59.476573 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:59.477100 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:59.477105 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:59.477413 (+ 308us) client_negotiation.cc:769] Sending connection context
0619 18:44:59.477592 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 18:44:59.477849 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":154,"threads_started":1}
I20260619 18:44:59.480384 3079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.143.125" port: 37115 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:44:59.480659 3004 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.143.125" port: 37115 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.124.keytab.
Entry for principal kudu/127.21.143.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.124.keytab.
I20260619 18:44:59.484609 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.124:43751
--webserver_interface=127.21.143.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.124.keytab
--principal=kudu/127.21.143.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:36191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:33747,127.21.143.125:37115,127.21.143.124:43751
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:44:59 dist-test-slave-mld5 krb5kdc[2923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894699, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for kudu/127.21.143.125@KRBTEST.COM
Jun 19 18:44:59 dist-test-slave-mld5 krb5kdc[2923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894699, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.125@KRBTEST.COM for kudu/127.21.143.125@KRBTEST.COM
I20260619 18:44:59.492422 3082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.480973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.125:37115 (local address 127.0.0.1:32890)
0619 18:44:59.481073 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:59.481088 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:59.481161 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:59.481924 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:59.481927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:59.482095 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:59.482208 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:59.482214 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.483831 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.483834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:59.484644 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:59.484655 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.485644 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.485650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:59.485815 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.486258 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:59.486278 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:59.488331 (+ 2053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:59.490786 (+ 2455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:59.490789 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:59.490791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:59.490968 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:59.491250 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:59.491255 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:59.491256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:59.491360 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:59.491892 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:59.491894 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:59.491979 (+ 85us) client_negotiation.cc:769] Sending connection context
0619 18:44:59.492092 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 18:44:59.492220 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":28,"spinlock_wait_cycles":1408}
I20260619 18:44:59.492456 3084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.481125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32890 (local address 127.21.143.125:37115)
0619 18:44:59.481552 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:59.481572 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:59.481582 (+ 10us) server_negotiation.cc:407] Connection header received
0619 18:44:59.481605 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:59.481607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:59.481780 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:59.481922 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:59.482322 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.483710 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.484772 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.485381 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.485493 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.488478 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:59.488481 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:59.488482 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:59.488500 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:59.490673 (+ 2173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.491074 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.491079 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.491081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.491159 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.491455 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.491460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.491462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.491789 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:59.491917 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:59.492186 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 18:44:59.492311 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":191,"server-negotiator.queue_time_us":377,"spinlock_wait_cycles":5632,"thread_start_us":114,"threads_started":1}
I20260619 18:44:59.492983 3006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.481011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.125:37115 (local address 127.0.0.1:32892)
0619 18:44:59.481188 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:59.481203 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:59.481277 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:59.481927 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:59.481930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:59.482095 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:59.482204 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:59.482211 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.483863 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.483869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:59.484601 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:59.484610 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.485437 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.485443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:59.485569 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.486081 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:59.486092 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:59.487903 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:59.491113 (+ 3210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:59.491119 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:59.491123 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:59.491450 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:59.491724 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:59.491731 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:59.491736 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:59.491841 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:59.492356 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:59.492361 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:59.492495 (+ 134us) client_negotiation.cc:769] Sending connection context
0619 18:44:59.492593 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 18:44:59.492755 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260619 18:44:59.493791 3085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.481045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32892 (local address 127.21.143.125:37115)
0619 18:44:59.481496 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:59.481507 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:59.481528 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:44:59.481572 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:59.481579 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:59.481780 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:59.481922 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:59.482322 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.483710 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.484718 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.485315 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.485821 (+ 506us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.488052 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:59.488062 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:59.488077 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:59.488101 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:59.490685 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.491551 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.491554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.491556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.491624 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.491971 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.491977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.491981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.492224 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:59.493305 (+ 1081us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:59.493405 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 18:44:59.493538 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":491,"server-negotiator.queue_time_us":361,"spinlock_wait_cycles":11008,"thread_start_us":145,"threads_started":1}
I20260619 18:44:59.495102 3004 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.143.124" port: 43751 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:44:59.495234 3079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.143.124" port: 43751 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:44:59.495896 3082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.495632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:43751 (local address 127.0.0.1:57822)
0619 18:44:59.495775 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:59.495844 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 18:44:59.495939 3006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.495632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:43751 (local address 127.0.0.1:57820)
0619 18:44:59.495783 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:59.495851 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 18:44:59.496076 3027 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.143.124:43751: connect: Connection refused (error 111)
I20260619 18:44:59.496803 3006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.496485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:43751 (local address 127.0.0.1:57832)
0619 18:44:59.496666 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:59.496728 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260619 18:44:59.496951 3004 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.143.124:43751: Network error: Client connection negotiation failed: client connection to 127.21.143.124:43751: connect: Connection refused (error 111)
I20260619 18:44:59.497169 3082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.496944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:43751 (local address 127.0.0.1:57838)
0619 18:44:59.497073 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:59.497121 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260619 18:44:59.497299 3079 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.143.124:43751: Network error: Client connection negotiation failed: client connection to 127.21.143.124:43751: connect: Connection refused (error 111)
I20260619 18:44:59.528138 3004 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.143.124" port: 43751 } attempt: 1
I20260619 18:44:59.529116 3006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.528769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:43751 (local address 127.0.0.1:57852)
0619 18:44:59.528966 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:59.529035 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 18:44:59.529987 3006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.529720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:43751 (local address 127.0.0.1:57854)
0619 18:44:59.529879 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:59.529930 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260619 18:44:59.530124 3004 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.143.124:43751: Network error: Client connection negotiation failed: client connection to 127.21.143.124:43751: connect: Connection refused (error 111)
I20260619 18:44:59.546604 3079 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.143.124" port: 43751 } attempt: 1
I20260619 18:44:59.547550 3082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.547209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:43751 (local address 127.0.0.1:57866)
0619 18:44:59.547399 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:59.547468 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 18:44:59.548297 3082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.548092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:43751 (local address 127.0.0.1:57878)
0619 18:44:59.548188 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:59.548234 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 18:44:59.548441 3079 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.143.124:43751: Network error: Client connection negotiation failed: client connection to 127.21.143.124:43751: connect: Connection refused (error 111)
W20260619 18:44:59.600832 3086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:59.601082 3086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:59.601121 3086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:59.605084 3086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:44:59.605141 3086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:59.605160 3086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:59.605208 3086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:44:59.605240 3086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:44:59.605289 3004 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.143.124" port: 43751 } attempt: 2
I20260619 18:44:59.606098 3006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.605813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:43751 (local address 127.0.0.1:57894)
0619 18:44:59.605952 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:59.606018 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 18:44:59.606864 3006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.606660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:43751 (local address 127.0.0.1:57898)
0619 18:44:59.606769 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:59.606812 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 18:44:59.607023 3004 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.143.124:43751: Network error: Client connection negotiation failed: client connection to 127.21.143.124:43751: connect: Connection refused (error 111)
I20260619 18:44:59.610664 3086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:36191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:33747,127.21.143.125:37115,127.21.143.124:43751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.124.keytab
--principal=kudu/127.21.143.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.124:43751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.143.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:44:59.611753 3086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:59.612730 3086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 18:44:59.616634 3079 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.143.124" port: 43751 } attempt: 2
I20260619 18:44:59.617327 3082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.617076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:43751 (local address 127.0.0.1:57912)
0619 18:44:59.617205 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:59.617262 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 18:44:59.618090 3082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.617882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:43751 (local address 127.0.0.1:57918)
0619 18:44:59.617990 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:59.618034 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 18:44:59.618280 3079 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.143.124:43751: Network error: Client connection negotiation failed: client connection to 127.21.143.124:43751: connect: Connection refused (error 111)
W20260619 18:44:59.621234 3093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:59.621258 3092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:59.621234 3095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:59.621588 3086 server_base.cc:1034] running on GCE node
I20260619 18:44:59.622208 3086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:59.623358 3086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:59.624506 3086 hybrid_clock.cc:648] HybridClock initialized: now 1781894699624490 us; error 26 us; skew 500 ppm
Jun 19 18:44:59 dist-test-slave-mld5 krb5kdc[2923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894699, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:59.627233 3086 init.cc:377] Logged in from keytab as kudu/127.21.143.124@KRBTEST.COM (short username kudu)
I20260619 18:44:59.628084 3086 webserver.cc:460] Webserver started at http://127.21.143.124:33147/ using document root <none> and password file <none>
I20260619 18:44:59.628587 3086 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:59.628640 3086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:44:59.628799 3086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:44:59.630729 3086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-2/data/instance:
uuid: "027b818e5df343bd9dfffdc755cc6da4"
format_stamp: "Formatted at 2026-06-19 18:44:59 on dist-test-slave-mld5"
server_key: "7772e63c173a6811a9fb33b0afc3ff53"
server_key_iv: "397df962c658411776cbfa82f8da757e"
server_key_version: "encryptionkey@0"
I20260619 18:44:59.631217 3086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-2/wal/instance:
uuid: "027b818e5df343bd9dfffdc755cc6da4"
format_stamp: "Formatted at 2026-06-19 18:44:59 on dist-test-slave-mld5"
server_key: "7772e63c173a6811a9fb33b0afc3ff53"
server_key_iv: "397df962c658411776cbfa82f8da757e"
server_key_version: "encryptionkey@0"
I20260619 18:44:59.634618 3086 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:44:59.636891 3102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:59.638057 3086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:44:59.638150 3086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-2/wal
uuid: "027b818e5df343bd9dfffdc755cc6da4"
format_stamp: "Formatted at 2026-06-19 18:44:59 on dist-test-slave-mld5"
server_key: "7772e63c173a6811a9fb33b0afc3ff53"
server_key_iv: "397df962c658411776cbfa82f8da757e"
server_key_version: "encryptionkey@0"
I20260619 18:44:59.638240 3086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:59.686173 3086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:59.689222 3086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:59.689376 3086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:59.696321 3086 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.124:43751
I20260619 18:44:59.696604 3154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.124:43751 every 8 connection(s)
I20260619 18:44:59.697418 3086 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-2/data/info.pb
I20260619 18:44:59.700320 3155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:44:59.702816 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 3086
I20260619 18:44:59.702946 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894542572984-22077-0/minicluster-data/master-2/wal/instance
I20260619 18:44:59.703251 22077 external_mini_cluster.cc:1428] Setting key 5d58cc163d10423b83d1199a85e9d579
I20260619 18:44:59.704545 3155 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33747 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:44:59.706188 3004 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.143.124" port: 43751 } attempt: 3
I20260619 18:44:59.709439 3079 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.143.124" port: 43751 } attempt: 3
Jun 19 18:44:59 dist-test-slave-mld5 krb5kdc[2923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894699, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for kudu/127.21.143.124@KRBTEST.COM
Jun 19 18:44:59 dist-test-slave-mld5 krb5kdc[2923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894699, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.124@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
Jun 19 18:44:59 dist-test-slave-mld5 krb5kdc[2923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.124@KRBTEST.COM
Jun 19 18:44:59 dist-test-slave-mld5 krb5kdc[2923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894699, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.125@KRBTEST.COM for kudu/127.21.143.124@KRBTEST.COM
I20260619 18:44:59.718761 3006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.706720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:43751 (local address 127.0.0.1:57938)
0619 18:44:59.706981 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:59.706995 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:59.707079 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:59.707777 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:59.707780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:59.707952 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:59.708030 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:59.708035 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.709137 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.709140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:59.709966 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:59.709976 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.711080 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.711084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:59.711170 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.711876 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:59.711890 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:59.713338 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:59.716504 (+ 3166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:59.716508 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:59.716510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:59.716893 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:59.717341 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:59.717347 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:59.717351 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:59.717453 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:59.718195 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:59.718198 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:59.718299 (+ 101us) client_negotiation.cc:769] Sending connection context
0619 18:44:59.718392 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 18:44:59.718538 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176}
I20260619 18:44:59.719003 3007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.706282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39042 (local address 127.21.143.126:33747)
0619 18:44:59.706518 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:59.706523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:59.706750 (+ 227us) server_negotiation.cc:407] Connection header received
0619 18:44:59.706964 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:59.706967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:59.707009 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:59.707050 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:59.708244 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.709021 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.710296 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.710614 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.710696 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.714855 (+ 4159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:59.714860 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:59.714862 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:59.714893 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:59.716872 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.717308 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.717314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.717317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.717403 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.717772 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.717776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.717778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.718083 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:59.718160 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:59.718705 (+ 545us) server_negotiation.cc:299] Negotiation successful
0619 18:44:59.718830 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"spinlock_wait_cycles":7424}
I20260619 18:44:59.720320 3082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.709845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:43751 (local address 127.0.0.1:57940)
0619 18:44:59.710074 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:59.710088 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:59.710169 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:59.710774 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:59.710780 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:59.710974 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:59.711089 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:59.711096 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.712247 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.712251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:59.713128 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:59.713143 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.713811 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.713815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:59.713921 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.714331 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:59.714345 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:59.715706 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:59.718283 (+ 2577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:59.718286 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:59.718288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:59.718520 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:59.719284 (+ 764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:59.719287 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:59.719289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:59.719415 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:59.719868 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:59.719871 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:59.719971 (+ 100us) client_negotiation.cc:769] Sending connection context
0619 18:44:59.720050 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 18:44:59.720172 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260619 18:44:59.720590 3159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.706231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33747 (local address 127.0.0.1:39042)
0619 18:44:59.706626 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:59.706675 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:59.706844 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:59.707255 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:59.707262 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:59.707726 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:59.708106 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:59.708119 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.709167 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.709169 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:59.710157 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:59.710166 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.710885 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.710888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:59.711052 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.711729 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:59.711755 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:59.714674 (+ 2919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:59.716961 (+ 2287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:59.716972 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:59.716981 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:59.717205 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:59.717516 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:59.717520 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:59.717522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:59.717656 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:59.718254 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:59.718263 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:59.718416 (+ 153us) client_negotiation.cc:769] Sending connection context
0619 18:44:59.719147 (+ 731us) client_negotiation.cc:240] Negotiation successful
0619 18:44:59.719628 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":202,"spinlock_wait_cycles":768,"thread_start_us":150,"threads_started":1}
I20260619 18:44:59.720588 3160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.707225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57938 (local address 127.21.143.124:43751)
0619 18:44:59.707555 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:59.707562 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:59.707571 (+ 9us) server_negotiation.cc:407] Connection header received
0619 18:44:59.707597 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:59.707599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:59.707655 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:59.707757 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:59.708146 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.709025 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.710169 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.710942 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.711415 (+ 473us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.713476 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:59.713483 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:59.713499 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:59.713519 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:59.716366 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.717031 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.717038 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.717041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.717216 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.717576 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.717580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.717582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.718079 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:59.719147 (+ 1068us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:59.719251 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 18:44:59.719543 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":160,"server-negotiator.queue_time_us":275,"spinlock_wait_cycles":7936,"thread_start_us":131,"threads_started":1}
I20260619 18:44:59.720588 3158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.704688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57934 (local address 127.21.143.124:43751)
0619 18:44:59.705087 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:59.705104 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:59.705139 (+ 35us) server_negotiation.cc:407] Connection header received
0619 18:44:59.705785 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:59.705803 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:59.706185 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:59.707417 (+ 1232us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:59.707843 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.708914 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.710110 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.710942 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.711627 (+ 685us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.715017 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:59.715023 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:59.715028 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:59.715526 (+ 498us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:59.718007 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.718524 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.718527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.718528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.718601 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.718950 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.718955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.718957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.719162 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:59.720018 (+ 856us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:59.720092 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 18:44:59.720190 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":121,"server-negotiator.queue_time_us":248,"thread_start_us":117,"threads_started":1}
I20260619 18:44:59.721463 3161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.710313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57940 (local address 127.21.143.124:43751)
0619 18:44:59.710528 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:59.710536 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:59.710547 (+ 11us) server_negotiation.cc:407] Connection header received
0619 18:44:59.710578 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:59.710581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:59.710620 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:59.711198 (+ 578us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:59.711340 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.712117 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.713277 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.713658 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.714125 (+ 467us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.715840 (+ 1715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:59.715843 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:59.715846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:59.715875 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:59.718170 (+ 2295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.718654 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.718657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.718658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.718753 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.719557 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.719560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.719561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.719748 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:59.721209 (+ 1461us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:59.721268 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 18:44:59.721348 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":85,"threads_started":1}
I20260619 18:44:59.722640 3155 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.143.125" port: 37115 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:44:59.725180 3079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 702d62e07157426ba4805408701b1863: Bootstrap starting.
I20260619 18:44:59.725555 3004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4148ad498ae04b82a8d4a935fad20f3d: Bootstrap starting.
I20260619 18:44:59.727988 3079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 702d62e07157426ba4805408701b1863: Neither blocks nor log segments found. Creating new log.
I20260619 18:44:59.727988 3004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4148ad498ae04b82a8d4a935fad20f3d: Neither blocks nor log segments found. Creating new log.
I20260619 18:44:59.728890 3079 log.cc:824] T 00000000000000000000000000000000 P 702d62e07157426ba4805408701b1863: Log is configured to *not* fsync() on all Append() calls
I20260619 18:44:59.728890 3004 log.cc:824] T 00000000000000000000000000000000 P 4148ad498ae04b82a8d4a935fad20f3d: Log is configured to *not* fsync() on all Append() calls
I20260619 18:44:59.731093 3079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 702d62e07157426ba4805408701b1863: No bootstrap required, opened a new log
Jun 19 18:44:59 dist-test-slave-mld5 krb5kdc[2923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894699, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.124@KRBTEST.COM for kudu/127.21.143.125@KRBTEST.COM
I20260619 18:44:59.733175 3004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4148ad498ae04b82a8d4a935fad20f3d: No bootstrap required, opened a new log
Jun 19 18:44:59 dist-test-slave-mld5 krb5kdc[2923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:44:59.736173 3085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.723256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32900 (local address 127.21.143.125:37115)
0619 18:44:59.723388 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:59.723392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:59.727577 (+ 4185us) server_negotiation.cc:407] Connection header received
0619 18:44:59.727692 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:59.727695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:59.727740 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:59.727820 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:59.728231 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.728758 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.729784 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.730181 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.730296 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.732606 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:59.732609 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:59.732612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:59.732635 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:59.734292 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.734777 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.734781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.734783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.734878 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.735185 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.735190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.735193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.735459 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:59.735538 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:59.735897 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 18:44:59.736020 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 18:44:59.736126 3079 raft_consensus.cc:348] T 00000000000000000000000000000000 P 702d62e07157426ba4805408701b1863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4148ad498ae04b82a8d4a935fad20f3d" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33747 } } peers { permanent_uuid: "702d62e07157426ba4805408701b1863" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 37115 } } peers { permanent_uuid: "027b818e5df343bd9dfffdc755cc6da4" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 43751 } }
I20260619 18:44:59.736678 3159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.723039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.125:37115 (local address 127.0.0.1:32900)
0619 18:44:59.727501 (+ 4462us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:59.727513 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:59.727608 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:59.727840 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:59.727843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:59.728005 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:59.728131 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:59.728140 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.728867 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.728872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:59.729632 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:59.729646 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.730318 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.730325 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:59.730428 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.730961 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:59.730975 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:59.732462 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:59.734419 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:59.734423 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:59.734425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:59.734666 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:59.734992 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:59.734996 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:59.735000 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:59.735084 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:59.735612 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:59.735615 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:59.735707 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 18:44:59.736370 (+ 663us) client_negotiation.cc:240] Negotiation successful
0619 18:44:59.736515 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4384}
I20260619 18:44:59.736671 3079 raft_consensus.cc:374] T 00000000000000000000000000000000 P 702d62e07157426ba4805408701b1863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:44:59.737185 3079 raft_consensus.cc:729] T 00000000000000000000000000000000 P 702d62e07157426ba4805408701b1863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 702d62e07157426ba4805408701b1863, State: Initialized, Role: FOLLOWER
I20260619 18:44:59.737386 3155 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.143.124" port: 43751 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:44:59.737963 3079 consensus_queue.cc:260] T 00000000000000000000000000000000 P 702d62e07157426ba4805408701b1863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4148ad498ae04b82a8d4a935fad20f3d" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33747 } } peers { permanent_uuid: "702d62e07157426ba4805408701b1863" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 37115 } } peers { permanent_uuid: "027b818e5df343bd9dfffdc755cc6da4" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 43751 } }
I20260619 18:44:59.737914 3004 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4148ad498ae04b82a8d4a935fad20f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4148ad498ae04b82a8d4a935fad20f3d" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33747 } } peers { permanent_uuid: "702d62e07157426ba4805408701b1863" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 37115 } } peers { permanent_uuid: "027b818e5df343bd9dfffdc755cc6da4" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 43751 } }
I20260619 18:44:59.738256 3004 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4148ad498ae04b82a8d4a935fad20f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:44:59.738330 3004 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4148ad498ae04b82a8d4a935fad20f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4148ad498ae04b82a8d4a935fad20f3d, State: Initialized, Role: FOLLOWER
I20260619 18:44:59.738539 3164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 702d62e07157426ba4805408701b1863 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4148ad498ae04b82a8d4a935fad20f3d" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33747 } } peers { permanent_uuid: "702d62e07157426ba4805408701b1863" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 37115 } } peers { permanent_uuid: "027b818e5df343bd9dfffdc755cc6da4" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 43751 } } }
I20260619 18:44:59.738751 3164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 702d62e07157426ba4805408701b1863 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:44:59.738866 3004 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4148ad498ae04b82a8d4a935fad20f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4148ad498ae04b82a8d4a935fad20f3d" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33747 } } peers { permanent_uuid: "702d62e07157426ba4805408701b1863" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 37115 } } peers { permanent_uuid: "027b818e5df343bd9dfffdc755cc6da4" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 43751 } }
I20260619 18:44:59.739423 3165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4148ad498ae04b82a8d4a935fad20f3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4148ad498ae04b82a8d4a935fad20f3d" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33747 } } peers { permanent_uuid: "702d62e07157426ba4805408701b1863" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 37115 } } peers { permanent_uuid: "027b818e5df343bd9dfffdc755cc6da4" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 43751 } } }
I20260619 18:44:59.739537 3079 sys_catalog.cc:564] T 00000000000000000000000000000000 P 702d62e07157426ba4805408701b1863 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:44:59.739632 3165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4148ad498ae04b82a8d4a935fad20f3d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:44:59.740159 3007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.729369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39048 (local address 127.21.143.126:33747)
0619 18:44:59.729494 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:59.729498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:59.729513 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:44:59.729547 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:59.729549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:59.729580 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:59.729674 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:59.730428 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.731084 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.732237 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.732545 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.732641 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.735934 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:59.735938 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:59.735941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:59.735966 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:59.737629 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.738466 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.738469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.738471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.738557 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.738880 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.738886 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.738888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.739102 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:59.739270 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:59.739916 (+ 646us) server_negotiation.cc:299] Negotiation successful
0619 18:44:59.740029 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 18:44:59.740510 3004 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4148ad498ae04b82a8d4a935fad20f3d [sys.catalog]: configured and running, proceeding with master startup.
W20260619 18:44:59.745548 3181 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 702d62e07157426ba4805408701b1863: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:44:59.745631 3181 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Jun 19 18:44:59 dist-test-slave-mld5 krb5kdc[2923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.125@KRBTEST.COM
Jun 19 18:44:59 dist-test-slave-mld5 krb5kdc[2923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894699, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.124@KRBTEST.COM for kudu/127.21.143.124@KRBTEST.COM
I20260619 18:44:59.747733 3011 master_runner.cc:186] Error getting master registration for 127.21.143.124:43751: OK, instance_id { permanent_uuid: "027b818e5df343bd9dfffdc755cc6da4" instance_seqno: 1781894699694321 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 18:44:59.751539 3011 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 18:44:59.753383 3085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.741697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32906 (local address 127.21.143.125:37115)
0619 18:44:59.741932 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:59.741936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:59.742110 (+ 174us) server_negotiation.cc:407] Connection header received
0619 18:44:59.742303 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:59.742306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:59.742355 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:59.742407 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:59.743341 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.744045 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.745072 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.745365 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.745437 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.748896 (+ 3459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:59.748901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:59.748907 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:59.748936 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:59.751130 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.751661 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.751665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.751668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.751784 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.752107 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.752110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.752111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.752311 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:59.752487 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:59.753118 (+ 631us) server_negotiation.cc:299] Negotiation successful
0619 18:44:59.753228 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260619 18:44:59.749116 2936 master_runner.cc:186] Error getting master registration for 127.21.143.124:43751: OK, instance_id { permanent_uuid: "027b818e5df343bd9dfffdc755cc6da4" instance_seqno: 1781894699694321 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 18:44:59.753664 2936 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 18:44:59.754052 3161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.737965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57948 (local address 127.21.143.124:43751)
0619 18:44:59.738074 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:59.738078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:59.740494 (+ 2416us) server_negotiation.cc:407] Connection header received
0619 18:44:59.740641 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:59.740644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:59.740679 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:59.740722 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:59.741306 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.742105 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.747358 (+ 5253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.747716 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.747828 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.750075 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:59.750078 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:59.750081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:59.750100 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:59.751684 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.752209 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.752212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.752214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.752283 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.752536 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.752540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.752543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.752742 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:59.752814 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:59.753242 (+ 428us) server_negotiation.cc:299] Negotiation successful
0619 18:44:59.753361 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 18:44:59.755641 3159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.737903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:43751 (local address 127.0.0.1:57948)
0619 18:44:59.740430 (+ 2527us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:59.740446 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:59.740551 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:59.740844 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:59.740848 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:59.741019 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:59.741171 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:59.741178 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.746537 (+ 5359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.746541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:59.747220 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:59.747229 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.747829 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.747833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:59.747928 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.748422 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:59.748436 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:59.749943 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:59.751884 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:59.751887 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:59.751890 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:59.752113 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:59.752375 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:59.752378 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:59.752380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:59.752454 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:59.752867 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:59.752871 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:59.752954 (+ 83us) client_negotiation.cc:769] Sending connection context
0619 18:44:59.755046 (+ 2092us) client_negotiation.cc:240] Negotiation successful
0619 18:44:59.755185 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2460}
W20260619 18:44:59.757050 3187 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4148ad498ae04b82a8d4a935fad20f3d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:44:59.757114 3187 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:44:59.759038 3155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 027b818e5df343bd9dfffdc755cc6da4: Bootstrap starting.
I20260619 18:44:59.760406 22077 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:44:59.765390 3155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 027b818e5df343bd9dfffdc755cc6da4: Neither blocks nor log segments found. Creating new log.
I20260619 18:44:59.766724 3155 log.cc:824] T 00000000000000000000000000000000 P 027b818e5df343bd9dfffdc755cc6da4: Log is configured to *not* fsync() on all Append() calls
I20260619 18:44:59.773384 3155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 027b818e5df343bd9dfffdc755cc6da4: No bootstrap required, opened a new log
I20260619 18:44:59.777724 3165 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4148ad498ae04b82a8d4a935fad20f3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:44:59.777951 3165 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4148ad498ae04b82a8d4a935fad20f3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4148ad498ae04b82a8d4a935fad20f3d" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33747 } } peers { permanent_uuid: "702d62e07157426ba4805408701b1863" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 37115 } } peers { permanent_uuid: "027b818e5df343bd9dfffdc755cc6da4" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 43751 } }
I20260619 18:44:59.778329 3085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.763472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32914 (local address 127.21.143.125:37115)
0619 18:44:59.763587 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:59.763592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:59.767550 (+ 3958us) server_negotiation.cc:407] Connection header received
0619 18:44:59.767740 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:59.767744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:59.767795 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:59.767849 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:59.768781 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.769531 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.770708 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.771051 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.771123 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.773330 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:59.773334 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:59.773337 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:59.773364 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:59.775332 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.775788 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.775791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.775793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.776143 (+ 350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.777126 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.777130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.777132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.777362 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:59.777459 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:59.778051 (+ 592us) server_negotiation.cc:299] Negotiation successful
0619 18:44:59.778167 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 18:44:59.778792 3155 raft_consensus.cc:348] T 00000000000000000000000000000000 P 027b818e5df343bd9dfffdc755cc6da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4148ad498ae04b82a8d4a935fad20f3d" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33747 } } peers { permanent_uuid: "702d62e07157426ba4805408701b1863" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 37115 } } peers { permanent_uuid: "027b818e5df343bd9dfffdc755cc6da4" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 43751 } }
I20260619 18:44:59.779034 3155 raft_consensus.cc:374] T 00000000000000000000000000000000 P 027b818e5df343bd9dfffdc755cc6da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:44:59.779042 3165 leader_election.cc:288] T 00000000000000000000000000000000 P 4148ad498ae04b82a8d4a935fad20f3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 702d62e07157426ba4805408701b1863 (127.21.143.125:37115), 027b818e5df343bd9dfffdc755cc6da4 (127.21.143.124:43751)
I20260619 18:44:59.779096 3155 raft_consensus.cc:729] T 00000000000000000000000000000000 P 027b818e5df343bd9dfffdc755cc6da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 027b818e5df343bd9dfffdc755cc6da4, State: Initialized, Role: FOLLOWER
I20260619 18:44:59.779616 3155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 027b818e5df343bd9dfffdc755cc6da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4148ad498ae04b82a8d4a935fad20f3d" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33747 } } peers { permanent_uuid: "702d62e07157426ba4805408701b1863" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 37115 } } peers { permanent_uuid: "027b818e5df343bd9dfffdc755cc6da4" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 43751 } }
I20260619 18:44:59.780006 3053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4148ad498ae04b82a8d4a935fad20f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "702d62e07157426ba4805408701b1863" is_pre_election: true
I20260619 18:44:59.780277 3053 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 702d62e07157426ba4805408701b1863 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4148ad498ae04b82a8d4a935fad20f3d in term 0.
I20260619 18:44:59.780664 2955 leader_election.cc:302] T 00000000000000000000000000000000 P 4148ad498ae04b82a8d4a935fad20f3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4148ad498ae04b82a8d4a935fad20f3d, 702d62e07157426ba4805408701b1863; no voters:
I20260619 18:44:59.781081 3155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 027b818e5df343bd9dfffdc755cc6da4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:44:59.781132 3129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4148ad498ae04b82a8d4a935fad20f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "027b818e5df343bd9dfffdc755cc6da4" is_pre_election: true
I20260619 18:44:59.781410 3129 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 027b818e5df343bd9dfffdc755cc6da4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4148ad498ae04b82a8d4a935fad20f3d in term 0.
I20260619 18:44:59.781716 3165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4148ad498ae04b82a8d4a935fad20f3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:44:59.781828 3165 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4148ad498ae04b82a8d4a935fad20f3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:44:59.781895 3165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4148ad498ae04b82a8d4a935fad20f3d [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:44:59.783360 3165 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4148ad498ae04b82a8d4a935fad20f3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4148ad498ae04b82a8d4a935fad20f3d" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33747 } } peers { permanent_uuid: "702d62e07157426ba4805408701b1863" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 37115 } } peers { permanent_uuid: "027b818e5df343bd9dfffdc755cc6da4" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 43751 } }
I20260619 18:44:59.784003 3165 leader_election.cc:288] T 00000000000000000000000000000000 P 4148ad498ae04b82a8d4a935fad20f3d [CANDIDATE]: Term 1 election: Requested vote from peers 702d62e07157426ba4805408701b1863 (127.21.143.125:37115), 027b818e5df343bd9dfffdc755cc6da4 (127.21.143.124:43751)
I20260619 18:44:59.784152 3053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4148ad498ae04b82a8d4a935fad20f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "702d62e07157426ba4805408701b1863"
I20260619 18:44:59.784276 3053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 702d62e07157426ba4805408701b1863 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:44:59.784310 3129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4148ad498ae04b82a8d4a935fad20f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "027b818e5df343bd9dfffdc755cc6da4"
I20260619 18:44:59.784415 3129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 027b818e5df343bd9dfffdc755cc6da4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:44:59.785593 3053 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 702d62e07157426ba4805408701b1863 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4148ad498ae04b82a8d4a935fad20f3d in term 1.
I20260619 18:44:59.785665 3129 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 027b818e5df343bd9dfffdc755cc6da4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4148ad498ae04b82a8d4a935fad20f3d in term 1.
I20260619 18:44:59.785962 2955 leader_election.cc:302] T 00000000000000000000000000000000 P 4148ad498ae04b82a8d4a935fad20f3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4148ad498ae04b82a8d4a935fad20f3d, 702d62e07157426ba4805408701b1863; no voters:
I20260619 18:44:59.786134 3165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4148ad498ae04b82a8d4a935fad20f3d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:44:59.786386 3165 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4148ad498ae04b82a8d4a935fad20f3d [term 1 LEADER]: Becoming Leader. State: Replica: 4148ad498ae04b82a8d4a935fad20f3d, State: Running, Role: LEADER
I20260619 18:44:59.786706 3165 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4148ad498ae04b82a8d4a935fad20f3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4148ad498ae04b82a8d4a935fad20f3d" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33747 } } peers { permanent_uuid: "702d62e07157426ba4805408701b1863" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 37115 } } peers { permanent_uuid: "027b818e5df343bd9dfffdc755cc6da4" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 43751 } }
I20260619 18:44:59.789158 3210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4148ad498ae04b82a8d4a935fad20f3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4148ad498ae04b82a8d4a935fad20f3d. Latest consensus state: current_term: 1 leader_uuid: "4148ad498ae04b82a8d4a935fad20f3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4148ad498ae04b82a8d4a935fad20f3d" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33747 } } peers { permanent_uuid: "702d62e07157426ba4805408701b1863" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 37115 } } peers { permanent_uuid: "027b818e5df343bd9dfffdc755cc6da4" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 43751 } } }
I20260619 18:44:59.789280 3210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4148ad498ae04b82a8d4a935fad20f3d [sys.catalog]: This master's current role is: LEADER
I20260619 18:44:59.789258 3086 master_runner.cc:263] No leader master found from master 027b818e5df343bd9dfffdc755cc6da4
I20260619 18:44:59.789330 3086 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 18:44:59.789638 3212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:44:59.790704 3212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:44:59.791164 3161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.763221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57950 (local address 127.21.143.124:43751)
0619 18:44:59.763372 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:59.763377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:59.765821 (+ 2444us) server_negotiation.cc:407] Connection header received
0619 18:44:59.765968 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:59.765971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:59.766026 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:59.766110 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:59.770253 (+ 4143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.771022 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.772081 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.772387 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.772461 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.775604 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:59.775608 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:59.775610 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:59.775635 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:59.777541 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.789736 (+ 12195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.789740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.789742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.789859 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.790167 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.790170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.790172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.790394 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:59.790514 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:59.790888 (+ 374us) server_negotiation.cc:299] Negotiation successful
0619 18:44:59.791016 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 18:44:59.792989 3007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.762883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39058 (local address 127.21.143.126:33747)
0619 18:44:59.763002 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:59.763007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:59.766153 (+ 3146us) server_negotiation.cc:407] Connection header received
0619 18:44:59.766330 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:59.766335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:59.766386 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:59.766438 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:59.768246 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.769060 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.772959 (+ 3899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.773298 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.773384 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.775990 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:59.775993 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:59.775996 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:59.776025 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:59.781590 (+ 5565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.791675 (+ 10085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.791679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.791681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.791769 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.792083 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.792086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.792089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.792295 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:59.792412 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:59.792750 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 18:44:59.792891 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 18:44:59.795771 3198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 027b818e5df343bd9dfffdc755cc6da4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4148ad498ae04b82a8d4a935fad20f3d" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33747 } } peers { permanent_uuid: "702d62e07157426ba4805408701b1863" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 37115 } } peers { permanent_uuid: "027b818e5df343bd9dfffdc755cc6da4" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 43751 } } }
I20260619 18:44:59.795905 3198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 027b818e5df343bd9dfffdc755cc6da4 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 18:44:59.799739 3209 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 027b818e5df343bd9dfffdc755cc6da4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:44:59.799829 3209 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:44:59.803486 3129 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 027b818e5df343bd9dfffdc755cc6da4 [term 1 FOLLOWER]: Refusing update from remote peer 4148ad498ae04b82a8d4a935fad20f3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:44:59.803910 3053 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 702d62e07157426ba4805408701b1863 [term 1 FOLLOWER]: Refusing update from remote peer 4148ad498ae04b82a8d4a935fad20f3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:44:59.804103 3210 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4148ad498ae04b82a8d4a935fad20f3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "027b818e5df343bd9dfffdc755cc6da4" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 43751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:44:59.804463 3165 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4148ad498ae04b82a8d4a935fad20f3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "702d62e07157426ba4805408701b1863" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 37115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:44:59.807835 3198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 027b818e5df343bd9dfffdc755cc6da4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4148ad498ae04b82a8d4a935fad20f3d. Latest consensus state: current_term: 1 leader_uuid: "4148ad498ae04b82a8d4a935fad20f3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4148ad498ae04b82a8d4a935fad20f3d" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33747 } } peers { permanent_uuid: "702d62e07157426ba4805408701b1863" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 37115 } } peers { permanent_uuid: "027b818e5df343bd9dfffdc755cc6da4" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 43751 } } }
I20260619 18:44:59.807960 3198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 027b818e5df343bd9dfffdc755cc6da4 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:44:59.808887 3164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 702d62e07157426ba4805408701b1863 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4148ad498ae04b82a8d4a935fad20f3d. Latest consensus state: current_term: 1 leader_uuid: "4148ad498ae04b82a8d4a935fad20f3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4148ad498ae04b82a8d4a935fad20f3d" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33747 } } peers { permanent_uuid: "702d62e07157426ba4805408701b1863" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 37115 } } peers { permanent_uuid: "027b818e5df343bd9dfffdc755cc6da4" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 43751 } } }
I20260619 18:44:59.809052 3164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 702d62e07157426ba4805408701b1863 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:44:59.816244 3210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4148ad498ae04b82a8d4a935fad20f3d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4148ad498ae04b82a8d4a935fad20f3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4148ad498ae04b82a8d4a935fad20f3d" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33747 } } peers { permanent_uuid: "702d62e07157426ba4805408701b1863" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 37115 } } peers { permanent_uuid: "027b818e5df343bd9dfffdc755cc6da4" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 43751 } } }
I20260619 18:44:59.816386 3210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4148ad498ae04b82a8d4a935fad20f3d [sys.catalog]: This master's current role is: LEADER
I20260619 18:44:59.816563 3164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 702d62e07157426ba4805408701b1863 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4148ad498ae04b82a8d4a935fad20f3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4148ad498ae04b82a8d4a935fad20f3d" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33747 } } peers { permanent_uuid: "702d62e07157426ba4805408701b1863" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 37115 } } peers { permanent_uuid: "027b818e5df343bd9dfffdc755cc6da4" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 43751 } } }
I20260619 18:44:59.816685 3164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 702d62e07157426ba4805408701b1863 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:44:59.816663 3213 mvcc.cc:205] Tried to move back new op lower bound from 7298640690387378176 to 7298640690329681920. Current Snapshot: MvccSnapshot[applied={T|T < 7298640690387378176}]
I20260619 18:44:59.819715 3217 mvcc.cc:205] Tried to move back new op lower bound from 7298640690387378176 to 7298640690329681920. Current Snapshot: MvccSnapshot[applied={T|T < 7298640690387378176}]
I20260619 18:44:59.820276 3165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4148ad498ae04b82a8d4a935fad20f3d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4148ad498ae04b82a8d4a935fad20f3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4148ad498ae04b82a8d4a935fad20f3d" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33747 } } peers { permanent_uuid: "702d62e07157426ba4805408701b1863" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 37115 } } peers { permanent_uuid: "027b818e5df343bd9dfffdc755cc6da4" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 43751 } } }
I20260619 18:44:59.820396 3165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4148ad498ae04b82a8d4a935fad20f3d [sys.catalog]: This master's current role is: LEADER
I20260619 18:44:59.821473 3198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 027b818e5df343bd9dfffdc755cc6da4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4148ad498ae04b82a8d4a935fad20f3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4148ad498ae04b82a8d4a935fad20f3d" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33747 } } peers { permanent_uuid: "702d62e07157426ba4805408701b1863" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 37115 } } peers { permanent_uuid: "027b818e5df343bd9dfffdc755cc6da4" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 43751 } } }
I20260619 18:44:59.821605 3198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 027b818e5df343bd9dfffdc755cc6da4 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:44:59.823668 3212 catalog_manager.cc:1347] Generated new cluster ID: ed33de15df7147d3a5ac9abbbc283b4b
I20260619 18:44:59.823726 3212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:44:59.823855 3215 mvcc.cc:205] Tried to move back new op lower bound from 7298640690387378176 to 7298640690329681920. Current Snapshot: MvccSnapshot[applied={T|T < 7298640690387378176}]
I20260619 18:44:59.850409 3212 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:44:59.854053 3212 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:44:59.879361 3212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4148ad498ae04b82a8d4a935fad20f3d: Generated new TSK 0
I20260619 18:44:59.880285 3212 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:45:00.746919 3181 catalog_manager.cc:1259] Loaded cluster ID: ed33de15df7147d3a5ac9abbbc283b4b
I20260619 18:45:00.747062 3181 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 702d62e07157426ba4805408701b1863: loading cluster ID for follower catalog manager: success
I20260619 18:45:00.748730 3181 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 702d62e07157426ba4805408701b1863: acquiring CA information for follower catalog manager: success
I20260619 18:45:00.749950 3181 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 702d62e07157426ba4805408701b1863: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 18:45:00.801187 3209 catalog_manager.cc:1259] Loaded cluster ID: ed33de15df7147d3a5ac9abbbc283b4b
I20260619 18:45:00.801280 3209 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 027b818e5df343bd9dfffdc755cc6da4: loading cluster ID for follower catalog manager: success
I20260619 18:45:00.803025 3209 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 027b818e5df343bd9dfffdc755cc6da4: acquiring CA information for follower catalog manager: success
I20260619 18:45:00.804296 3209 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 027b818e5df343bd9dfffdc755cc6da4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 18:45:00.953636 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 2936
I20260619 18:45:00.960572 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 3011
I20260619 18:45:00.966938 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 3086
2026-06-19T18:45:00Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3103 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:45:01 dist-test-slave-mld5 krb5kdc[3230](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:45:01 dist-test-slave-mld5 krb5kdc[3230](info): set up 2 sockets
Jun 19 18:45:01 dist-test-slave-mld5 krb5kdc[3230](info): commencing operation
krb5kdc: starting...
W20260619 18:45:02.015666 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:45:02 dist-test-slave-mld5 krb5kdc[3230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:45:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:45:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.143.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
I20260619 18:45:02.136447 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:41491
--webserver_interface=127.21.143.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:40097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:41491
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:45:02.250739 3243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:02.251042 3243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:02.251084 3243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:02.255195 3243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:45:02.255267 3243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:02.255288 3243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:02.255324 3243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:45:02.255340 3243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:45:02.260784 3243 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:40097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:41491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:41491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:45:02.261896 3243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:02.262954 3243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 18:45:02.271658 3243 server_base.cc:1034] running on GCE node
W20260619 18:45:02.271605 3251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:02.271605 3249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:02.271617 3248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:02.272560 3243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:02.273654 3243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:02.274845 3243 hybrid_clock.cc:648] HybridClock initialized: now 1781894702274830 us; error 35 us; skew 500 ppm
Jun 19 18:45:02 dist-test-slave-mld5 krb5kdc[3230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894702, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:45:02.277763 3243 init.cc:377] Logged in from keytab as kudu/127.21.143.126@KRBTEST.COM (short username kudu)
I20260619 18:45:02.278708 3243 webserver.cc:460] Webserver started at http://127.21.143.126:41303/ using document root <none> and password file <none>
I20260619 18:45:02.279337 3243 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:02.279451 3243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:02.279695 3243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:45:02.281772 3243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "92f976a0228f4f8181602fb7f247d350"
format_stamp: "Formatted at 2026-06-19 18:45:02 on dist-test-slave-mld5"
server_key: "d6d4475272c1d5cfa642bbfe9401f837"
server_key_iv: "ed19bec6864d4ca5aea49912ee71102b"
server_key_version: "encryptionkey@0"
I20260619 18:45:02.282392 3243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "92f976a0228f4f8181602fb7f247d350"
format_stamp: "Formatted at 2026-06-19 18:45:02 on dist-test-slave-mld5"
server_key: "d6d4475272c1d5cfa642bbfe9401f837"
server_key_iv: "ed19bec6864d4ca5aea49912ee71102b"
server_key_version: "encryptionkey@0"
I20260619 18:45:02.286144 3243 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 18:45:02.288656 3258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:02.289906 3243 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:45:02.290066 3243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "92f976a0228f4f8181602fb7f247d350"
format_stamp: "Formatted at 2026-06-19 18:45:02 on dist-test-slave-mld5"
server_key: "d6d4475272c1d5cfa642bbfe9401f837"
server_key_iv: "ed19bec6864d4ca5aea49912ee71102b"
server_key_version: "encryptionkey@0"
I20260619 18:45:02.290189 3243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:02.307380 3243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:02.310320 3243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:02.310544 3243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:02.317996 3243 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:41491
I20260619 18:45:02.318006 3310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:41491 every 8 connection(s)
I20260619 18:45:02.318981 3243 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:45:02.321815 3311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:02.322700 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 3243
I20260619 18:45:02.322830 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:45:02.323146 22077 external_mini_cluster.cc:1428] Setting key fcfe6d7858ebffe58c6891d4be2bd21d
I20260619 18:45:02.327788 3311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92f976a0228f4f8181602fb7f247d350: Bootstrap starting.
I20260619 18:45:02.329936 3311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92f976a0228f4f8181602fb7f247d350: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:02.330890 3311 log.cc:824] T 00000000000000000000000000000000 P 92f976a0228f4f8181602fb7f247d350: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:45:02 dist-test-slave-mld5 krb5kdc[3230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:45:02.333209 3311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92f976a0228f4f8181602fb7f247d350: No bootstrap required, opened a new log
I20260619 18:45:02.337452 3314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:02.324524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53242 (local address 127.21.143.126:41491)
0619 18:45:02.324974 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:02.324989 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:02.325022 (+ 33us) server_negotiation.cc:407] Connection header received
0619 18:45:02.325640 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:02.325655 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:02.325989 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:02.326323 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:02.327111 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:02.328189 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:02.329017 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:02.329480 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:02.329649 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:02.332512 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:02.332517 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:02.332531 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:02.332557 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:02.334340 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:02.334837 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:02.334842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:02.334847 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:02.334981 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:02.335348 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:02.335353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:02.335358 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:02.335757 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:02.335930 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:02.336259 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 18:45:02.336629 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":138,"threads_started":1}
I20260619 18:45:02.337428 3311 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92f976a0228f4f8181602fb7f247d350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92f976a0228f4f8181602fb7f247d350" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 41491 } }
I20260619 18:45:02.337747 3311 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92f976a0228f4f8181602fb7f247d350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:02.337855 3311 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92f976a0228f4f8181602fb7f247d350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92f976a0228f4f8181602fb7f247d350, State: Initialized, Role: FOLLOWER
I20260619 18:45:02.338317 3311 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92f976a0228f4f8181602fb7f247d350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92f976a0228f4f8181602fb7f247d350" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 41491 } }
I20260619 18:45:02.338477 3311 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92f976a0228f4f8181602fb7f247d350 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:45:02.338557 3311 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92f976a0228f4f8181602fb7f247d350 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:45:02.338652 3311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92f976a0228f4f8181602fb7f247d350 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:45:02.339926 3311 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92f976a0228f4f8181602fb7f247d350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92f976a0228f4f8181602fb7f247d350" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 41491 } }
I20260619 18:45:02.340351 3311 leader_election.cc:302] T 00000000000000000000000000000000 P 92f976a0228f4f8181602fb7f247d350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92f976a0228f4f8181602fb7f247d350; no voters:
I20260619 18:45:02.340752 3311 leader_election.cc:288] T 00000000000000000000000000000000 P 92f976a0228f4f8181602fb7f247d350 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:45:02.340869 22077 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:45:02.340862 3316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92f976a0228f4f8181602fb7f247d350 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:45:02.341058 3316 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92f976a0228f4f8181602fb7f247d350 [term 1 LEADER]: Becoming Leader. State: Replica: 92f976a0228f4f8181602fb7f247d350, State: Running, Role: LEADER
I20260619 18:45:02.341403 3316 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92f976a0228f4f8181602fb7f247d350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92f976a0228f4f8181602fb7f247d350" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 41491 } }
I20260619 18:45:02.342104 3311 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92f976a0228f4f8181602fb7f247d350 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:45:02.342697 3318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92f976a0228f4f8181602fb7f247d350 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92f976a0228f4f8181602fb7f247d350. Latest consensus state: current_term: 1 leader_uuid: "92f976a0228f4f8181602fb7f247d350" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92f976a0228f4f8181602fb7f247d350" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 41491 } } }
I20260619 18:45:02.342716 3317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92f976a0228f4f8181602fb7f247d350 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92f976a0228f4f8181602fb7f247d350" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92f976a0228f4f8181602fb7f247d350" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 41491 } } }
I20260619 18:45:02.342856 3317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92f976a0228f4f8181602fb7f247d350 [sys.catalog]: This master's current role is: LEADER
I20260619 18:45:02.342878 3318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92f976a0228f4f8181602fb7f247d350 [sys.catalog]: This master's current role is: LEADER
I20260619 18:45:02.343266 3322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:45:02.346753 3322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:45:02.352284 3322 catalog_manager.cc:1347] Generated new cluster ID: 283e76d893bf4e09b51a9da6afb5ed9d
I20260619 18:45:02.352382 3322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:45:02.368603 3322 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:45:02.369648 3322 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:45:02.376925 3322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92f976a0228f4f8181602fb7f247d350: Generated new TSK 0
I20260619 18:45:02.377713 3322 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:45:02.397293 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 3243
2026-06-19T18:45:02Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1427 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:45:02 dist-test-slave-mld5 krb5kdc[3338](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:45:02 dist-test-slave-mld5 krb5kdc[3338](info): set up 2 sockets
Jun 19 18:45:02 dist-test-slave-mld5 krb5kdc[3338](info): commencing operation
krb5kdc: starting...
W20260619 18:45:03.435561 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:45:03 dist-test-slave-mld5 krb5kdc[3338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:45:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:45:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.143.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
I20260619 18:45:03.549935 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:42911
--webserver_interface=127.21.143.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:42911
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:45:03.658840 3351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:03.659152 3351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:03.659225 3351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:03.663218 3351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:45:03.663305 3351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:03.663365 3351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:03.663422 3351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:45:03.663460 3351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:45:03.668380 3351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:42911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:42911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:45:03.669562 3351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:03.670663 3351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:03.678602 3359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:03.678660 3357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:03.678853 3356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:03.679085 3351 server_base.cc:1034] running on GCE node
I20260619 18:45:03.679652 3351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:03.680636 3351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:03.681792 3351 hybrid_clock.cc:648] HybridClock initialized: now 1781894703681777 us; error 29 us; skew 500 ppm
Jun 19 18:45:03 dist-test-slave-mld5 krb5kdc[3338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894703, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:45:03.689509 3351 init.cc:377] Logged in from keytab as kudu/127.21.143.126@KRBTEST.COM (short username kudu)
I20260619 18:45:03.690325 3351 webserver.cc:460] Webserver started at http://127.21.143.126:43643/ using document root <none> and password file <none>
I20260619 18:45:03.690799 3351 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:03.690852 3351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:03.691001 3351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:45:03.692963 3351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "46b26682e2a74a828401a0e226e8ef74"
format_stamp: "Formatted at 2026-06-19 18:45:03 on dist-test-slave-mld5"
server_key: "2f215579542df4a63e9c2fd0b94af197"
server_key_iv: "6d6d23e3d59d005d35f09eb3e3075348"
server_key_version: "encryptionkey@0"
I20260619 18:45:03.693444 3351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "46b26682e2a74a828401a0e226e8ef74"
format_stamp: "Formatted at 2026-06-19 18:45:03 on dist-test-slave-mld5"
server_key: "2f215579542df4a63e9c2fd0b94af197"
server_key_iv: "6d6d23e3d59d005d35f09eb3e3075348"
server_key_version: "encryptionkey@0"
I20260619 18:45:03.697023 3351 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:45:03.699244 3366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:03.700415 3351 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:45:03.700562 3351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "46b26682e2a74a828401a0e226e8ef74"
format_stamp: "Formatted at 2026-06-19 18:45:03 on dist-test-slave-mld5"
server_key: "2f215579542df4a63e9c2fd0b94af197"
server_key_iv: "6d6d23e3d59d005d35f09eb3e3075348"
server_key_version: "encryptionkey@0"
I20260619 18:45:03.700665 3351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:03.712783 3351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:03.716017 3351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:03.716214 3351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:03.723145 3351 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:42911
I20260619 18:45:03.723423 3418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:42911 every 8 connection(s)
I20260619 18:45:03.724184 3351 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:45:03.725844 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 3351
I20260619 18:45:03.725986 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:45:03.726316 22077 external_mini_cluster.cc:1428] Setting key 050b7f537e07de8c14b605fa9360dbbd
I20260619 18:45:03.727092 3419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:03.733842 3419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74: Bootstrap starting.
Jun 19 18:45:03 dist-test-slave-mld5 krb5kdc[3338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:45:03.736182 3419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:03.737139 3419 log.cc:824] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:03.739387 3419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74: No bootstrap required, opened a new log
I20260619 18:45:03.741056 3422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:03.727695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34538 (local address 127.21.143.126:42911)
0619 18:45:03.728099 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:03.728118 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:03.728159 (+ 41us) server_negotiation.cc:407] Connection header received
0619 18:45:03.729009 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:03.729035 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:03.729439 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:03.729815 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:03.730500 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:03.731986 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:03.732821 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:03.733496 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:03.733648 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:03.736134 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:03.736142 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:03.736160 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:03.736192 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:03.738044 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:03.738445 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:03.738457 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:03.738463 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:03.738604 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:03.738858 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:03.738861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:03.738863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:03.739370 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:03.739569 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:03.739830 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 18:45:03.740116 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":110,"threads_started":1}
I20260619 18:45:03.744043 3419 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b26682e2a74a828401a0e226e8ef74" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42911 } }
I20260619 18:45:03.744251 3419 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:03.744292 3419 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46b26682e2a74a828401a0e226e8ef74, State: Initialized, Role: FOLLOWER
I20260619 18:45:03.744643 3419 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b26682e2a74a828401a0e226e8ef74" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42911 } }
I20260619 18:45:03.744750 3419 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:45:03.744792 3419 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:45:03.744859 3419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:45:03.745723 3419 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b26682e2a74a828401a0e226e8ef74" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42911 } }
I20260619 18:45:03.746183 3419 leader_election.cc:302] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46b26682e2a74a828401a0e226e8ef74; no voters:
I20260619 18:45:03.746534 3419 leader_election.cc:288] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:45:03.746857 3424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:45:03.747061 3424 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [term 1 LEADER]: Becoming Leader. State: Replica: 46b26682e2a74a828401a0e226e8ef74, State: Running, Role: LEADER
I20260619 18:45:03.747437 3424 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b26682e2a74a828401a0e226e8ef74" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42911 } }
I20260619 18:45:03.747817 3419 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:45:03.749001 3426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46b26682e2a74a828401a0e226e8ef74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b26682e2a74a828401a0e226e8ef74" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42911 } } }
I20260619 18:45:03.749125 3426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [sys.catalog]: This master's current role is: LEADER
I20260619 18:45:03.749521 3433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:45:03.749475 3425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46b26682e2a74a828401a0e226e8ef74. Latest consensus state: current_term: 1 leader_uuid: "46b26682e2a74a828401a0e226e8ef74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b26682e2a74a828401a0e226e8ef74" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42911 } } }
I20260619 18:45:03.749586 3425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [sys.catalog]: This master's current role is: LEADER
I20260619 18:45:03.752413 3433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:45:03.757709 3433 catalog_manager.cc:1347] Generated new cluster ID: a26e46602b274f61b30907f91272a69d
I20260619 18:45:03.757776 3433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:45:03.774837 3433 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:45:03.775907 3433 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:45:03.780520 3433 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74: Generated new TSK 0
I20260619 18:45:03.781224 3433 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.143.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab.
Entry for principal kudu/127.21.143.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab.
I20260619 18:45:03.811285 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.65:0
--local_ip_for_outbound_sockets=127.21.143.65
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:42911
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:45:03.920498 3445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:03.920735 3445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:03.920812 3445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:03.924860 3445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:03.924916 3445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:03.925006 3445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.65
I20260619 18:45:03.930202 3445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:42911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.65
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:45:03.931356 3445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:03.932420 3445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:03.943475 3453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:03.943485 3450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:03.943485 3451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:03.943557 3445 server_base.cc:1034] running on GCE node
I20260619 18:45:03.944224 3445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:03.944852 3445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:03.946017 3445 hybrid_clock.cc:648] HybridClock initialized: now 1781894703946006 us; error 29 us; skew 500 ppm
Jun 19 18:45:03 dist-test-slave-mld5 krb5kdc[3338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894703, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:45:03.948793 3445 init.cc:377] Logged in from keytab as kudu/127.21.143.65@KRBTEST.COM (short username kudu)
I20260619 18:45:03.949695 3445 webserver.cc:460] Webserver started at http://127.21.143.65:38075/ using document root <none> and password file <none>
I20260619 18:45:03.950224 3445 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:03.950282 3445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:03.950445 3445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:45:03.952489 3445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/data/instance:
uuid: "9d8f9ac072db4db8aba2735fd7613aa0"
format_stamp: "Formatted at 2026-06-19 18:45:03 on dist-test-slave-mld5"
server_key: "bbb16c96d2ea28fd3f9622a935c0d0d3"
server_key_iv: "c5122a1d8301dc408d4cad3cbc084877"
server_key_version: "encryptionkey@0"
I20260619 18:45:03.953027 3445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance:
uuid: "9d8f9ac072db4db8aba2735fd7613aa0"
format_stamp: "Formatted at 2026-06-19 18:45:03 on dist-test-slave-mld5"
server_key: "bbb16c96d2ea28fd3f9622a935c0d0d3"
server_key_iv: "c5122a1d8301dc408d4cad3cbc084877"
server_key_version: "encryptionkey@0"
I20260619 18:45:03.956724 3445 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:45:03.958946 3460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:03.960178 3445 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:45:03.960304 3445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/wal
uuid: "9d8f9ac072db4db8aba2735fd7613aa0"
format_stamp: "Formatted at 2026-06-19 18:45:03 on dist-test-slave-mld5"
server_key: "bbb16c96d2ea28fd3f9622a935c0d0d3"
server_key_iv: "c5122a1d8301dc408d4cad3cbc084877"
server_key_version: "encryptionkey@0"
I20260619 18:45:03.960404 3445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:03.988150 3445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:03.991714 3445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:03.991925 3445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:03.992552 3445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:45:03.993404 3445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:45:03.993449 3445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:03.993517 3445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:45:03.993548 3445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:04.002867 3445 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.65:42281
I20260619 18:45:04.002938 3573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.65:42281 every 8 connection(s)
I20260619 18:45:04.003954 3445 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
I20260619 18:45:04.007503 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 3445
I20260619 18:45:04.007649 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance
I20260619 18:45:04.007941 22077 external_mini_cluster.cc:1428] Setting key 919b46bcf8c002d715bc08831feafaf9
Jun 19 18:45:04 dist-test-slave-mld5 krb5kdc[3338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894703, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.143.66@KRBTEST.COM; defaulting to no policy
I20260619 18:45:04.018901 3422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:04.005818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:42269 (local address 127.21.143.126:42911)
0619 18:45:04.005955 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:04.005958 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:04.006763 (+ 805us) server_negotiation.cc:407] Connection header received
0619 18:45:04.008437 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:04.008440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:04.008519 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:04.008607 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:04.010072 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:04.010614 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:04.011888 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:04.012142 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:04.012258 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:04.015077 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:04.015080 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:04.015082 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:04.015105 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:04.016725 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:04.017234 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:04.017238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:04.017240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:04.017365 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:04.017739 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:04.017744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:04.017746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:04.017939 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:04.018057 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:04.018577 (+ 520us) server_negotiation.cc:299] Negotiation successful
0619 18:45:04.018721 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.66@KRBTEST.COM" created.
I20260619 18:45:04.019971 3576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:04.006172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.65:42269)
0619 18:45:04.006615 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:04.006660 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:04.008192 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:04.008835 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:04.008844 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:04.009354 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:45:04.009912 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:04.009926 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:04.010774 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:04.010778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:04.011760 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:04.011770 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:04.012298 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:04.012302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:04.012479 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:04.013162 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:45:04.013178 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:45:04.014898 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:45:04.016837 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:04.016843 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:04.016859 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:04.017127 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:04.017456 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:04.017460 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:04.017462 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:04.017615 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:04.018055 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:45:04.018063 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:45:04.018330 (+ 267us) client_negotiation.cc:769] Sending connection context
0619 18:45:04.018534 (+ 204us) client_negotiation.cc:240] Negotiation successful
0619 18:45:04.018881 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":110,"threads_started":1}
I20260619 18:45:04.021385 3574 heartbeater.cc:347] Connected to a master server at 127.21.143.126:42911
I20260619 18:45:04.021669 3574 heartbeater.cc:464] Registering TS with master...
I20260619 18:45:04.022179 3574 heartbeater.cc:511] Master 127.21.143.126:42911 requested a full tablet report, sending...
I20260619 18:45:04.023890 3383 ts_manager.cc:194] Registered new tserver with Master: 9d8f9ac072db4db8aba2735fd7613aa0 (127.21.143.65:42281)
I20260619 18:45:04.025714 3383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.65@KRBTEST.COM'} at 127.21.143.65:42269
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab.
Entry for principal kudu/127.21.143.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab.
I20260619 18:45:04.034448 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.66:0
--local_ip_for_outbound_sockets=127.21.143.66
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:42911
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:45:04.143955 3579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:04.144233 3579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:04.144276 3579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:04.148458 3579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:04.148530 3579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:04.148639 3579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.66
I20260619 18:45:04.154138 3579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:42911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.66
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:45:04.155272 3579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:04.156366 3579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:04.166394 3585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:04.166400 3587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:04.166630 3579 server_base.cc:1034] running on GCE node
W20260619 18:45:04.166405 3584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:04.167186 3579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:04.167912 3579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:04.169113 3579 hybrid_clock.cc:648] HybridClock initialized: now 1781894704169076 us; error 51 us; skew 500 ppm
Jun 19 18:45:04 dist-test-slave-mld5 krb5kdc[3338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894704, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:45:04.172313 3579 init.cc:377] Logged in from keytab as kudu/127.21.143.66@KRBTEST.COM (short username kudu)
I20260619 18:45:04.173213 3579 webserver.cc:460] Webserver started at http://127.21.143.66:45415/ using document root <none> and password file <none>
I20260619 18:45:04.173800 3579 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:04.173893 3579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:04.174139 3579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:45:04.176304 3579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/data/instance:
uuid: "7e0eabab97f74bceaf95d7d08feea40c"
format_stamp: "Formatted at 2026-06-19 18:45:04 on dist-test-slave-mld5"
server_key: "67da6db9e2e65f66ac4ae634a9d3eb57"
server_key_iv: "fd5a7ee1d9a78ca5f054b06d94db421a"
server_key_version: "encryptionkey@0"
I20260619 18:45:04.176878 3579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance:
uuid: "7e0eabab97f74bceaf95d7d08feea40c"
format_stamp: "Formatted at 2026-06-19 18:45:04 on dist-test-slave-mld5"
server_key: "67da6db9e2e65f66ac4ae634a9d3eb57"
server_key_iv: "fd5a7ee1d9a78ca5f054b06d94db421a"
server_key_version: "encryptionkey@0"
I20260619 18:45:04.180862 3579 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:45:04.183393 3594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:04.184676 3579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:45:04.184819 3579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/wal
uuid: "7e0eabab97f74bceaf95d7d08feea40c"
format_stamp: "Formatted at 2026-06-19 18:45:04 on dist-test-slave-mld5"
server_key: "67da6db9e2e65f66ac4ae634a9d3eb57"
server_key_iv: "fd5a7ee1d9a78ca5f054b06d94db421a"
server_key_version: "encryptionkey@0"
I20260619 18:45:04.184940 3579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:04.201099 3579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:04.204106 3579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:04.204315 3579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:04.204937 3579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:45:04.205803 3579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:45:04.205875 3579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:04.205946 3579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:45:04.206002 3579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:04.215708 3579 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.66:44607
I20260619 18:45:04.215731 3707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.66:44607 every 8 connection(s)
I20260619 18:45:04.216730 3579 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
I20260619 18:45:04.220752 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 3579
I20260619 18:45:04.220860 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance
I20260619 18:45:04.221153 22077 external_mini_cluster.cc:1428] Setting key 4df04793c8cc754c8660cc1e83f9c17d
Jun 19 18:45:04 dist-test-slave-mld5 krb5kdc[3338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894704, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.143.67@KRBTEST.COM; defaulting to no policy
I20260619 18:45:04.231467 3422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:04.218684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:43769 (local address 127.21.143.126:42911)
0619 18:45:04.218831 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:04.218836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:04.219853 (+ 1017us) server_negotiation.cc:407] Connection header received
0619 18:45:04.221424 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:04.221429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:04.221497 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:04.221599 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:04.223073 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:04.223953 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:04.225114 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:04.225341 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:04.225459 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:04.227918 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:04.227921 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:04.227924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:04.227956 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:04.229364 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:04.229891 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:04.229895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:04.229897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:04.229978 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:04.230314 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:04.230319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:04.230321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:04.230553 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:04.230655 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:04.231145 (+ 490us) server_negotiation.cc:299] Negotiation successful
0619 18:45:04.231269 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 18:45:04.232347 3710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:04.219153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.66:43769)
0619 18:45:04.219702 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:04.219746 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:04.221176 (+ 1430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:04.221808 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:04.221815 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:04.222318 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:45:04.222927 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:04.222941 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:04.224070 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:04.224073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:04.224987 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:04.224995 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:04.225502 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:04.225506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:04.225681 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:04.226279 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:45:04.226295 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:45:04.227750 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:45:04.229504 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:04.229510 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:04.229524 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:04.229781 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:04.230075 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:04.230078 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:04.230080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:04.230221 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:04.230663 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:45:04.230672 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:45:04.230910 (+ 238us) client_negotiation.cc:769] Sending connection context
0619 18:45:04.231105 (+ 195us) client_negotiation.cc:240] Negotiation successful
0619 18:45:04.231485 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":188,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.67@KRBTEST.COM" created.
I20260619 18:45:04.233515 3708 heartbeater.cc:347] Connected to a master server at 127.21.143.126:42911
I20260619 18:45:04.233776 3708 heartbeater.cc:464] Registering TS with master...
I20260619 18:45:04.234278 3708 heartbeater.cc:511] Master 127.21.143.126:42911 requested a full tablet report, sending...
I20260619 18:45:04.235355 3383 ts_manager.cc:194] Registered new tserver with Master: 7e0eabab97f74bceaf95d7d08feea40c (127.21.143.66:44607)
I20260619 18:45:04.235934 3383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.66@KRBTEST.COM'} at 127.21.143.66:43769
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab.
Entry for principal kudu/127.21.143.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab.
I20260619 18:45:04.246133 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.67:0
--local_ip_for_outbound_sockets=127.21.143.67
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:42911
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:45:04.356832 3713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:04.357123 3713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:04.357168 3713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:04.361163 3713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:04.361232 3713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:04.361292 3713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.67
I20260619 18:45:04.367132 3713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:42911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.67
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:45:04.368294 3713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:04.369326 3713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:04.379081 3718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:04.379105 3719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:04.379081 3721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:04.379204 3713 server_base.cc:1034] running on GCE node
I20260619 18:45:04.379812 3713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:04.380432 3713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:04.381614 3713 hybrid_clock.cc:648] HybridClock initialized: now 1781894704381596 us; error 40 us; skew 500 ppm
Jun 19 18:45:04 dist-test-slave-mld5 krb5kdc[3338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894704, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:45:04.384267 3713 init.cc:377] Logged in from keytab as kudu/127.21.143.67@KRBTEST.COM (short username kudu)
I20260619 18:45:04.385058 3713 webserver.cc:460] Webserver started at http://127.21.143.67:37647/ using document root <none> and password file <none>
I20260619 18:45:04.385591 3713 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:04.385658 3713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:04.385823 3713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:45:04.388069 3713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/data/instance:
uuid: "b3d01ca389bf4a4fa503d849a23b71c6"
format_stamp: "Formatted at 2026-06-19 18:45:04 on dist-test-slave-mld5"
server_key: "51a67d461aecc2e01b9b9770862d2372"
server_key_iv: "3dd4c89f810e944bad8e5da56f0653bc"
server_key_version: "encryptionkey@0"
I20260619 18:45:04.388626 3713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance:
uuid: "b3d01ca389bf4a4fa503d849a23b71c6"
format_stamp: "Formatted at 2026-06-19 18:45:04 on dist-test-slave-mld5"
server_key: "51a67d461aecc2e01b9b9770862d2372"
server_key_iv: "3dd4c89f810e944bad8e5da56f0653bc"
server_key_version: "encryptionkey@0"
I20260619 18:45:04.392611 3713 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:45:04.395093 3728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:04.396561 3713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:45:04.396663 3713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/wal
uuid: "b3d01ca389bf4a4fa503d849a23b71c6"
format_stamp: "Formatted at 2026-06-19 18:45:04 on dist-test-slave-mld5"
server_key: "51a67d461aecc2e01b9b9770862d2372"
server_key_iv: "3dd4c89f810e944bad8e5da56f0653bc"
server_key_version: "encryptionkey@0"
I20260619 18:45:04.396760 3713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:04.410635 3713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:04.413647 3713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:04.413800 3713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:04.414340 3713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:45:04.415118 3713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:45:04.415163 3713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:04.415201 3713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:45:04.415217 3713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:04.424042 3713 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.67:39183
I20260619 18:45:04.424064 3841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.67:39183 every 8 connection(s)
I20260619 18:45:04.424944 3713 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
I20260619 18:45:04.432807 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 3713
I20260619 18:45:04.433018 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance
I20260619 18:45:04.433421 22077 external_mini_cluster.cc:1428] Setting key 7b8c576c30c6e8ca31b1bd5aac070958
Jun 19 18:45:04 dist-test-slave-mld5 krb5kdc[3338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894704, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:45:04.439162 3422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:04.426706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:51559 (local address 127.21.143.126:42911)
0619 18:45:04.426870 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:04.426876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:04.427756 (+ 880us) server_negotiation.cc:407] Connection header received
0619 18:45:04.429102 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:04.429105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:04.429152 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:04.429225 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:04.430433 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:04.430997 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:04.432134 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:04.432358 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:04.432483 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:04.435496 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:04.435500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:04.435502 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:04.435534 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:04.437044 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:04.437586 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:04.437590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:04.437592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:04.437682 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:04.438036 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:04.438040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:04.438042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:04.438241 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:04.438345 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:04.438869 (+ 524us) server_negotiation.cc:299] Negotiation successful
0619 18:45:04.438999 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 18:45:04.440071 3844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:04.427099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.67:51559)
0619 18:45:04.427603 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:04.427648 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:04.428902 (+ 1254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:04.429393 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:04.429400 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:04.429770 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:45:04.430266 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:04.430280 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:04.431111 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:04.431113 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:04.432021 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:04.432030 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:04.432544 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:04.432548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:04.432760 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:04.433444 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:45:04.433470 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:45:04.435282 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:45:04.437169 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:04.437176 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:04.437194 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:04.437476 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:04.437775 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:04.437779 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:04.437781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:04.437936 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:04.438353 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:45:04.438362 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:45:04.438624 (+ 262us) client_negotiation.cc:769] Sending connection context
0619 18:45:04.438821 (+ 197us) client_negotiation.cc:240] Negotiation successful
0619 18:45:04.439180 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":167,"threads_started":1}
I20260619 18:45:04.441203 3842 heartbeater.cc:347] Connected to a master server at 127.21.143.126:42911
I20260619 18:45:04.441504 3842 heartbeater.cc:464] Registering TS with master...
I20260619 18:45:04.442022 3842 heartbeater.cc:511] Master 127.21.143.126:42911 requested a full tablet report, sending...
I20260619 18:45:04.443089 3383 ts_manager.cc:194] Registered new tserver with Master: b3d01ca389bf4a4fa503d849a23b71c6 (127.21.143.67:39183)
I20260619 18:45:04.443722 3383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.67@KRBTEST.COM'} at 127.21.143.67:51559
I20260619 18:45:04.447549 22077 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:45:04.456395 3422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:04.449616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34548 (local address 127.21.143.126:42911)
0619 18:45:04.449736 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:04.449739 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:04.449938 (+ 199us) server_negotiation.cc:407] Connection header received
0619 18:45:04.450043 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:04.450045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:04.450084 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:04.450147 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:04.450794 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:04.451343 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:04.452113 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:04.452319 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:04.452393 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:04.453618 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:04.453621 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:04.453624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:04.453645 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:04.454839 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:04.455212 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:04.455215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:04.455216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:04.455283 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:04.455610 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:04.455613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:04.455614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:04.455860 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:04.455975 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:04.456167 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 18:45:04.456272 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 18:45:04.459187 3383 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:45:04.462342 3383 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:45:04.477439 3855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:04.471091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60692 (local address 127.21.143.65:42281)
0619 18:45:04.472322 (+ 1231us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:04.472336 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:04.472360 (+ 24us) server_negotiation.cc:407] Connection header received
0619 18:45:04.472440 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:04.472444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:04.472675 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:04.472833 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:04.473341 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:04.474464 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:04.476133 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:04.476706 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:04.476834 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:04.477030 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:04.477151 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 18:45:04.477270 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1121,"thread_start_us":55,"threads_started":1}
I20260619 18:45:04.477452 3854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:04.470979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:42281 (local address 127.0.0.1:60692)
0619 18:45:04.471836 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:04.471895 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:04.472037 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:04.473019 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:04.473024 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:04.473047 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:04.473206 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:04.473215 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:04.474616 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:04.474619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:04.476023 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:04.476029 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:04.476836 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:04.476840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:04.476958 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:04.476997 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 18:45:04.477120 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 18:45:04.477254 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":82,"spinlock_wait_cycles":3200,"thread_start_us":82,"threads_started":1}
I20260619 18:45:04.477484 3853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:04.471033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:44607 (local address 127.0.0.1:48320)
0619 18:45:04.471836 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:04.471895 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:04.472037 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:04.472828 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:04.472834 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:04.472858 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:04.473006 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:04.473015 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:04.474538 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:04.474544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:04.475822 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:04.475832 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:04.476916 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:04.476920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:04.477008 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:04.477029 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 18:45:04.477090 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 18:45:04.477280 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":28,"spinlock_wait_cycles":7680,"thread_start_us":95,"threads_started":1}
I20260619 18:45:04.477811 3858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:04.472042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45268 (local address 127.21.143.67:39183)
0619 18:45:04.472667 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:04.472676 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:04.472694 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:45:04.472774 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:04.472781 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:04.472968 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:04.473128 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:04.473373 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:04.474124 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:04.475691 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:04.476782 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:04.476893 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:04.477034 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:04.477553 (+ 519us) server_negotiation.cc:299] Negotiation successful
0619 18:45:04.477663 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":119,"threads_started":1}
I20260619 18:45:04.478111 3856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:04.471531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48320 (local address 127.21.143.66:44607)
0619 18:45:04.472323 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:04.472336 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:04.472360 (+ 24us) server_negotiation.cc:407] Connection header received
0619 18:45:04.472439 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:04.472444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:04.472661 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:04.472833 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:04.473144 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:04.474371 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:04.475955 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:04.476773 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:04.477660 (+ 887us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:04.477800 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:04.477852 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 18:45:04.477948 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":117,"threads_started":1}
I20260619 18:45:04.477484 3857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:04.471727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:39183 (local address 127.0.0.1:45268)
0619 18:45:04.472419 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:04.472432 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:04.472525 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:04.473101 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:04.473104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:04.473115 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:04.473253 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:04.473263 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:04.474256 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:04.474262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:04.475568 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:04.475577 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:04.477094 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:04.477098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:04.477186 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:04.477205 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 18:45:04.477295 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 18:45:04.477361 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":37,"thread_start_us":80,"threads_started":1}
I20260619 18:45:04.481258 3776 tablet_service.cc:1467] Processing CreateTablet for tablet d739e0c47a344284b637d47a0fc11f57 (DEFAULT_TABLE table=test-table [id=5a6323b44dfe4190b05d46e9ab6b429f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:45:04.481221 3508 tablet_service.cc:1467] Processing CreateTablet for tablet d739e0c47a344284b637d47a0fc11f57 (DEFAULT_TABLE table=test-table [id=5a6323b44dfe4190b05d46e9ab6b429f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:45:04.481397 3642 tablet_service.cc:1467] Processing CreateTablet for tablet d739e0c47a344284b637d47a0fc11f57 (DEFAULT_TABLE table=test-table [id=5a6323b44dfe4190b05d46e9ab6b429f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:45:04.482353 3642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d739e0c47a344284b637d47a0fc11f57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:04.482363 3776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d739e0c47a344284b637d47a0fc11f57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:04.482661 3508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d739e0c47a344284b637d47a0fc11f57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:04.488871 3859 tablet_bootstrap.cc:492] T d739e0c47a344284b637d47a0fc11f57 P 7e0eabab97f74bceaf95d7d08feea40c: Bootstrap starting.
I20260619 18:45:04.490338 3861 tablet_bootstrap.cc:492] T d739e0c47a344284b637d47a0fc11f57 P b3d01ca389bf4a4fa503d849a23b71c6: Bootstrap starting.
I20260619 18:45:04.490463 3860 tablet_bootstrap.cc:492] T d739e0c47a344284b637d47a0fc11f57 P 9d8f9ac072db4db8aba2735fd7613aa0: Bootstrap starting.
I20260619 18:45:04.491743 3859 tablet_bootstrap.cc:654] T d739e0c47a344284b637d47a0fc11f57 P 7e0eabab97f74bceaf95d7d08feea40c: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:04.492635 3860 tablet_bootstrap.cc:654] T d739e0c47a344284b637d47a0fc11f57 P 9d8f9ac072db4db8aba2735fd7613aa0: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:04.492635 3861 tablet_bootstrap.cc:654] T d739e0c47a344284b637d47a0fc11f57 P b3d01ca389bf4a4fa503d849a23b71c6: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:04.492616 3859 log.cc:824] T d739e0c47a344284b637d47a0fc11f57 P 7e0eabab97f74bceaf95d7d08feea40c: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:04.493974 3860 log.cc:824] T d739e0c47a344284b637d47a0fc11f57 P 9d8f9ac072db4db8aba2735fd7613aa0: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:04.493974 3861 log.cc:824] T d739e0c47a344284b637d47a0fc11f57 P b3d01ca389bf4a4fa503d849a23b71c6: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:04.494760 3859 tablet_bootstrap.cc:492] T d739e0c47a344284b637d47a0fc11f57 P 7e0eabab97f74bceaf95d7d08feea40c: No bootstrap required, opened a new log
I20260619 18:45:04.494979 3859 ts_tablet_manager.cc:1397] T d739e0c47a344284b637d47a0fc11f57 P 7e0eabab97f74bceaf95d7d08feea40c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 18:45:04.496032 3861 tablet_bootstrap.cc:492] T d739e0c47a344284b637d47a0fc11f57 P b3d01ca389bf4a4fa503d849a23b71c6: No bootstrap required, opened a new log
I20260619 18:45:04.496212 3861 ts_tablet_manager.cc:1397] T d739e0c47a344284b637d47a0fc11f57 P b3d01ca389bf4a4fa503d849a23b71c6: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 18:45:04.496248 3860 tablet_bootstrap.cc:492] T d739e0c47a344284b637d47a0fc11f57 P 9d8f9ac072db4db8aba2735fd7613aa0: No bootstrap required, opened a new log
I20260619 18:45:04.496408 3860 ts_tablet_manager.cc:1397] T d739e0c47a344284b637d47a0fc11f57 P 9d8f9ac072db4db8aba2735fd7613aa0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 18:45:04.498275 3859 raft_consensus.cc:348] T d739e0c47a344284b637d47a0fc11f57 P 7e0eabab97f74bceaf95d7d08feea40c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d8f9ac072db4db8aba2735fd7613aa0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 42281 } } peers { permanent_uuid: "7e0eabab97f74bceaf95d7d08feea40c" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 44607 } } peers { permanent_uuid: "b3d01ca389bf4a4fa503d849a23b71c6" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 39183 } }
I20260619 18:45:04.498525 3859 raft_consensus.cc:374] T d739e0c47a344284b637d47a0fc11f57 P 7e0eabab97f74bceaf95d7d08feea40c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:04.498603 3859 raft_consensus.cc:729] T d739e0c47a344284b637d47a0fc11f57 P 7e0eabab97f74bceaf95d7d08feea40c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e0eabab97f74bceaf95d7d08feea40c, State: Initialized, Role: FOLLOWER
I20260619 18:45:04.499071 3859 consensus_queue.cc:260] T d739e0c47a344284b637d47a0fc11f57 P 7e0eabab97f74bceaf95d7d08feea40c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d8f9ac072db4db8aba2735fd7613aa0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 42281 } } peers { permanent_uuid: "7e0eabab97f74bceaf95d7d08feea40c" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 44607 } } peers { permanent_uuid: "b3d01ca389bf4a4fa503d849a23b71c6" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 39183 } }
I20260619 18:45:04.500141 3861 raft_consensus.cc:348] T d739e0c47a344284b637d47a0fc11f57 P b3d01ca389bf4a4fa503d849a23b71c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d8f9ac072db4db8aba2735fd7613aa0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 42281 } } peers { permanent_uuid: "7e0eabab97f74bceaf95d7d08feea40c" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 44607 } } peers { permanent_uuid: "b3d01ca389bf4a4fa503d849a23b71c6" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 39183 } }
I20260619 18:45:04.500108 3860 raft_consensus.cc:348] T d739e0c47a344284b637d47a0fc11f57 P 9d8f9ac072db4db8aba2735fd7613aa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d8f9ac072db4db8aba2735fd7613aa0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 42281 } } peers { permanent_uuid: "7e0eabab97f74bceaf95d7d08feea40c" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 44607 } } peers { permanent_uuid: "b3d01ca389bf4a4fa503d849a23b71c6" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 39183 } }
I20260619 18:45:04.500415 3860 raft_consensus.cc:374] T d739e0c47a344284b637d47a0fc11f57 P 9d8f9ac072db4db8aba2735fd7613aa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:04.500473 3861 raft_consensus.cc:374] T d739e0c47a344284b637d47a0fc11f57 P b3d01ca389bf4a4fa503d849a23b71c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:04.500516 3860 raft_consensus.cc:729] T d739e0c47a344284b637d47a0fc11f57 P 9d8f9ac072db4db8aba2735fd7613aa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d8f9ac072db4db8aba2735fd7613aa0, State: Initialized, Role: FOLLOWER
I20260619 18:45:04.500566 3861 raft_consensus.cc:729] T d739e0c47a344284b637d47a0fc11f57 P b3d01ca389bf4a4fa503d849a23b71c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3d01ca389bf4a4fa503d849a23b71c6, State: Initialized, Role: FOLLOWER
I20260619 18:45:04.500689 3859 ts_tablet_manager.cc:1428] T d739e0c47a344284b637d47a0fc11f57 P 7e0eabab97f74bceaf95d7d08feea40c: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260619 18:45:04.501067 3708 heartbeater.cc:503] Master 127.21.143.126:42911 was elected leader, sending a full tablet report...
I20260619 18:45:04.501036 3860 consensus_queue.cc:260] T d739e0c47a344284b637d47a0fc11f57 P 9d8f9ac072db4db8aba2735fd7613aa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d8f9ac072db4db8aba2735fd7613aa0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 42281 } } peers { permanent_uuid: "7e0eabab97f74bceaf95d7d08feea40c" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 44607 } } peers { permanent_uuid: "b3d01ca389bf4a4fa503d849a23b71c6" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 39183 } }
I20260619 18:45:04.501036 3861 consensus_queue.cc:260] T d739e0c47a344284b637d47a0fc11f57 P b3d01ca389bf4a4fa503d849a23b71c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d8f9ac072db4db8aba2735fd7613aa0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 42281 } } peers { permanent_uuid: "7e0eabab97f74bceaf95d7d08feea40c" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 44607 } } peers { permanent_uuid: "b3d01ca389bf4a4fa503d849a23b71c6" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 39183 } }
I20260619 18:45:04.501888 3842 heartbeater.cc:503] Master 127.21.143.126:42911 was elected leader, sending a full tablet report...
I20260619 18:45:04.502058 3860 ts_tablet_manager.cc:1428] T d739e0c47a344284b637d47a0fc11f57 P 9d8f9ac072db4db8aba2735fd7613aa0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 18:45:04.502059 3861 ts_tablet_manager.cc:1428] T d739e0c47a344284b637d47a0fc11f57 P b3d01ca389bf4a4fa503d849a23b71c6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 18:45:04.502310 3574 heartbeater.cc:503] Master 127.21.143.126:42911 was elected leader, sending a full tablet report...
W20260619 18:45:04.505237 3575 tablet.cc:2369] T d739e0c47a344284b637d47a0fc11f57 P 9d8f9ac072db4db8aba2735fd7613aa0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:45:04.614760 3865 raft_consensus.cc:482] T d739e0c47a344284b637d47a0fc11f57 P 7e0eabab97f74bceaf95d7d08feea40c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:45:04.614967 3865 raft_consensus.cc:504] T d739e0c47a344284b637d47a0fc11f57 P 7e0eabab97f74bceaf95d7d08feea40c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d8f9ac072db4db8aba2735fd7613aa0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 42281 } } peers { permanent_uuid: "7e0eabab97f74bceaf95d7d08feea40c" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 44607 } } peers { permanent_uuid: "b3d01ca389bf4a4fa503d849a23b71c6" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 39183 } }
I20260619 18:45:04.616313 3865 leader_election.cc:288] T d739e0c47a344284b637d47a0fc11f57 P 7e0eabab97f74bceaf95d7d08feea40c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d8f9ac072db4db8aba2735fd7613aa0 (127.21.143.65:42281), b3d01ca389bf4a4fa503d849a23b71c6 (127.21.143.67:39183)
I20260619 18:45:04.621465 3710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:04.616545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:42281 (local address 127.21.143.66:60449)
0619 18:45:04.616729 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:04.616745 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:04.616883 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:04.617154 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:04.617159 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:04.617186 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:04.617311 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:04.617319 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:04.618387 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:04.618390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:04.620022 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:04.620033 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:04.621055 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:04.621058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:04.621165 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:04.621194 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 18:45:04.621272 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 18:45:04.621326 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":164}
I20260619 18:45:04.621484 3858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:04.616723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:59795 (local address 127.21.143.67:39183)
0619 18:45:04.616851 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:04.616857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:04.617085 (+ 228us) server_negotiation.cc:407] Connection header received
0619 18:45:04.617171 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:04.617185 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:04.617241 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:04.617310 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:04.617573 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:04.618285 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:04.620154 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:04.620765 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:04.620881 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:04.620948 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:04.621267 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 18:45:04.621360 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 18:45:04.621484 3855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:04.616598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:60449 (local address 127.21.143.65:42281)
0619 18:45:04.616755 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:04.616761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:04.616900 (+ 139us) server_negotiation.cc:407] Connection header received
0619 18:45:04.616969 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:04.616972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:04.617016 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:04.617120 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:04.617439 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:04.618255 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:04.620280 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:04.620855 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:04.620961 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:04.621009 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:04.621300 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 18:45:04.621374 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 18:45:04.621474 3868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:04.616551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:39183 (local address 127.21.143.66:59795)
0619 18:45:04.616972 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:04.616985 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:04.617072 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:04.617334 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:04.617337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:04.617354 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:04.617474 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:04.617478 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:04.618387 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:04.618390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:04.620022 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:04.620033 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:04.621010 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:04.621014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:04.621123 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:04.621148 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 18:45:04.621231 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 18:45:04.621310 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":212,"thread_start_us":130,"threads_started":1}
I20260619 18:45:04.622097 3528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d739e0c47a344284b637d47a0fc11f57" candidate_uuid: "7e0eabab97f74bceaf95d7d08feea40c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d8f9ac072db4db8aba2735fd7613aa0" is_pre_election: true
I20260619 18:45:04.622141 3796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d739e0c47a344284b637d47a0fc11f57" candidate_uuid: "7e0eabab97f74bceaf95d7d08feea40c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d01ca389bf4a4fa503d849a23b71c6" is_pre_election: true
I20260619 18:45:04.622431 3528 raft_consensus.cc:2454] T d739e0c47a344284b637d47a0fc11f57 P 9d8f9ac072db4db8aba2735fd7613aa0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e0eabab97f74bceaf95d7d08feea40c in term 0.
I20260619 18:45:04.622457 3796 raft_consensus.cc:2454] T d739e0c47a344284b637d47a0fc11f57 P b3d01ca389bf4a4fa503d849a23b71c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e0eabab97f74bceaf95d7d08feea40c in term 0.
I20260619 18:45:04.622920 3598 leader_election.cc:302] T d739e0c47a344284b637d47a0fc11f57 P 7e0eabab97f74bceaf95d7d08feea40c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e0eabab97f74bceaf95d7d08feea40c, b3d01ca389bf4a4fa503d849a23b71c6; no voters:
I20260619 18:45:04.623206 3865 raft_consensus.cc:2781] T d739e0c47a344284b637d47a0fc11f57 P 7e0eabab97f74bceaf95d7d08feea40c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:45:04.623286 3865 raft_consensus.cc:482] T d739e0c47a344284b637d47a0fc11f57 P 7e0eabab97f74bceaf95d7d08feea40c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:45:04.623384 3865 raft_consensus.cc:3037] T d739e0c47a344284b637d47a0fc11f57 P 7e0eabab97f74bceaf95d7d08feea40c [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:45:04.624411 3865 raft_consensus.cc:504] T d739e0c47a344284b637d47a0fc11f57 P 7e0eabab97f74bceaf95d7d08feea40c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d8f9ac072db4db8aba2735fd7613aa0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 42281 } } peers { permanent_uuid: "7e0eabab97f74bceaf95d7d08feea40c" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 44607 } } peers { permanent_uuid: "b3d01ca389bf4a4fa503d849a23b71c6" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 39183 } }
I20260619 18:45:04.624784 3865 leader_election.cc:288] T d739e0c47a344284b637d47a0fc11f57 P 7e0eabab97f74bceaf95d7d08feea40c [CANDIDATE]: Term 1 election: Requested vote from peers 9d8f9ac072db4db8aba2735fd7613aa0 (127.21.143.65:42281), b3d01ca389bf4a4fa503d849a23b71c6 (127.21.143.67:39183)
I20260619 18:45:04.625118 3528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d739e0c47a344284b637d47a0fc11f57" candidate_uuid: "7e0eabab97f74bceaf95d7d08feea40c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d8f9ac072db4db8aba2735fd7613aa0"
I20260619 18:45:04.625147 3796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d739e0c47a344284b637d47a0fc11f57" candidate_uuid: "7e0eabab97f74bceaf95d7d08feea40c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d01ca389bf4a4fa503d849a23b71c6"
I20260619 18:45:04.625238 3796 raft_consensus.cc:3037] T d739e0c47a344284b637d47a0fc11f57 P b3d01ca389bf4a4fa503d849a23b71c6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:45:04.625238 3528 raft_consensus.cc:3037] T d739e0c47a344284b637d47a0fc11f57 P 9d8f9ac072db4db8aba2735fd7613aa0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:45:04.626412 3528 raft_consensus.cc:2454] T d739e0c47a344284b637d47a0fc11f57 P 9d8f9ac072db4db8aba2735fd7613aa0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e0eabab97f74bceaf95d7d08feea40c in term 1.
I20260619 18:45:04.626412 3796 raft_consensus.cc:2454] T d739e0c47a344284b637d47a0fc11f57 P b3d01ca389bf4a4fa503d849a23b71c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e0eabab97f74bceaf95d7d08feea40c in term 1.
I20260619 18:45:04.626668 3595 leader_election.cc:302] T d739e0c47a344284b637d47a0fc11f57 P 7e0eabab97f74bceaf95d7d08feea40c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e0eabab97f74bceaf95d7d08feea40c, 9d8f9ac072db4db8aba2735fd7613aa0; no voters:
I20260619 18:45:04.626827 3865 raft_consensus.cc:2781] T d739e0c47a344284b637d47a0fc11f57 P 7e0eabab97f74bceaf95d7d08feea40c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:45:04.627038 3865 raft_consensus.cc:686] T d739e0c47a344284b637d47a0fc11f57 P 7e0eabab97f74bceaf95d7d08feea40c [term 1 LEADER]: Becoming Leader. State: Replica: 7e0eabab97f74bceaf95d7d08feea40c, State: Running, Role: LEADER
I20260619 18:45:04.627400 3865 consensus_queue.cc:237] T d739e0c47a344284b637d47a0fc11f57 P 7e0eabab97f74bceaf95d7d08feea40c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d8f9ac072db4db8aba2735fd7613aa0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 42281 } } peers { permanent_uuid: "7e0eabab97f74bceaf95d7d08feea40c" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 44607 } } peers { permanent_uuid: "b3d01ca389bf4a4fa503d849a23b71c6" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 39183 } }
I20260619 18:45:04.630353 3381 catalog_manager.cc:5495] T d739e0c47a344284b637d47a0fc11f57 P 7e0eabab97f74bceaf95d7d08feea40c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e0eabab97f74bceaf95d7d08feea40c (127.21.143.66). New cstate: current_term: 1 leader_uuid: "7e0eabab97f74bceaf95d7d08feea40c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d8f9ac072db4db8aba2735fd7613aa0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 42281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e0eabab97f74bceaf95d7d08feea40c" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 44607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3d01ca389bf4a4fa503d849a23b71c6" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 39183 } health_report { overall_health: UNKNOWN } } }
I20260619 18:45:04.666132 3856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:04.662399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48328 (local address 127.21.143.66:44607)
0619 18:45:04.662583 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:04.662590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:04.662606 (+ 16us) server_negotiation.cc:407] Connection header received
0619 18:45:04.662699 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:04.662701 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:04.662757 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:04.662891 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:45:04.663149 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:04.663760 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:04.664664 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:04.664876 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:04.664990 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:04.665309 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:45:04.665735 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:45:04.665876 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:04.665910 (+ 34us) server_negotiation.cc:299] Negotiation successful
0619 18:45:04.665967 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 18:45:04.672159 3528 raft_consensus.cc:1261] T d739e0c47a344284b637d47a0fc11f57 P 9d8f9ac072db4db8aba2735fd7613aa0 [term 1 FOLLOWER]: Refusing update from remote peer 7e0eabab97f74bceaf95d7d08feea40c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:45:04.672159 3796 raft_consensus.cc:1261] T d739e0c47a344284b637d47a0fc11f57 P b3d01ca389bf4a4fa503d849a23b71c6 [term 1 FOLLOWER]: Refusing update from remote peer 7e0eabab97f74bceaf95d7d08feea40c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:45:04.672905 3865 consensus_queue.cc:1035] T d739e0c47a344284b637d47a0fc11f57 P 7e0eabab97f74bceaf95d7d08feea40c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d01ca389bf4a4fa503d849a23b71c6" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:45:04.673111 3869 consensus_queue.cc:1035] T d739e0c47a344284b637d47a0fc11f57 P 7e0eabab97f74bceaf95d7d08feea40c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d8f9ac072db4db8aba2735fd7613aa0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 42281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 18:45:04.675824 3843 tablet.cc:2369] T d739e0c47a344284b637d47a0fc11f57 P b3d01ca389bf4a4fa503d849a23b71c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:45:04.682610 3872 mvcc.cc:205] Tried to move back new op lower bound from 7298640710329860096 to 7298640710156554240. Current Snapshot: MvccSnapshot[applied={T|T < 7298640710329860096 or (T in {7298640710329860096})}]
I20260619 18:45:04.682812 3873 mvcc.cc:205] Tried to move back new op lower bound from 7298640710329860096 to 7298640710156554240. Current Snapshot: MvccSnapshot[applied={T|T < 7298640710329860096}]
I20260619 18:45:04.684453 3875 mvcc.cc:205] Tried to move back new op lower bound from 7298640710329860096 to 7298640710156554240. Current Snapshot: MvccSnapshot[applied={T|T < 7298640710329860096}]
I20260619 18:45:04.689724 3381 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:45:04.689949 3381 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:45:04.692400 3381 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74: Sending DeleteTablet for 3 replicas of tablet d739e0c47a344284b637d47a0fc11f57
I20260619 18:45:04.693231 3508 tablet_service.cc:1514] Processing DeleteTablet for tablet d739e0c47a344284b637d47a0fc11f57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:45:04 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:60692
I20260619 18:45:04.693387 3776 tablet_service.cc:1514] Processing DeleteTablet for tablet d739e0c47a344284b637d47a0fc11f57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:45:04 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:45268
I20260619 18:45:04.693387 3642 tablet_service.cc:1514] Processing DeleteTablet for tablet d739e0c47a344284b637d47a0fc11f57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:45:04 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:48320
I20260619 18:45:04.693889 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 3445
I20260619 18:45:04.693893 3881 tablet_replica.cc:330] T d739e0c47a344284b637d47a0fc11f57 P 9d8f9ac072db4db8aba2735fd7613aa0: stopping tablet replica
I20260619 18:45:04.694226 3881 raft_consensus.cc:2229] T d739e0c47a344284b637d47a0fc11f57 P 9d8f9ac072db4db8aba2735fd7613aa0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:45:04.694615 3882 tablet_replica.cc:330] T d739e0c47a344284b637d47a0fc11f57 P 7e0eabab97f74bceaf95d7d08feea40c: stopping tablet replica
I20260619 18:45:04.694897 3882 raft_consensus.cc:2229] T d739e0c47a344284b637d47a0fc11f57 P 7e0eabab97f74bceaf95d7d08feea40c [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:45:04.695341 3882 raft_consensus.cc:2258] T d739e0c47a344284b637d47a0fc11f57 P 7e0eabab97f74bceaf95d7d08feea40c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:45:04.695721 3883 tablet_replica.cc:330] T d739e0c47a344284b637d47a0fc11f57 P b3d01ca389bf4a4fa503d849a23b71c6: stopping tablet replica
I20260619 18:45:04.696004 3883 raft_consensus.cc:2229] T d739e0c47a344284b637d47a0fc11f57 P b3d01ca389bf4a4fa503d849a23b71c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:45:04.696305 3883 raft_consensus.cc:2258] T d739e0c47a344284b637d47a0fc11f57 P b3d01ca389bf4a4fa503d849a23b71c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:45:04.697892 3882 ts_tablet_manager.cc:1905] T d739e0c47a344284b637d47a0fc11f57 P 7e0eabab97f74bceaf95d7d08feea40c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:45:04.700608 3882 ts_tablet_manager.cc:1918] T d739e0c47a344284b637d47a0fc11f57 P 7e0eabab97f74bceaf95d7d08feea40c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:45:04.700850 3882 log.cc:1196] T d739e0c47a344284b637d47a0fc11f57 P 7e0eabab97f74bceaf95d7d08feea40c: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/wal/wals/d739e0c47a344284b637d47a0fc11f57
I20260619 18:45:04.701859 3883 ts_tablet_manager.cc:1905] T d739e0c47a344284b637d47a0fc11f57 P b3d01ca389bf4a4fa503d849a23b71c6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 18:45:04.701911 3367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.143.65:42281 (error 108)
I20260619 18:45:04.702150 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 3579
I20260619 18:45:04.705610 3883 ts_tablet_manager.cc:1918] T d739e0c47a344284b637d47a0fc11f57 P b3d01ca389bf4a4fa503d849a23b71c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:45:04.705729 3883 log.cc:1196] T d739e0c47a344284b637d47a0fc11f57 P b3d01ca389bf4a4fa503d849a23b71c6: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/wal/wals/d739e0c47a344284b637d47a0fc11f57
I20260619 18:45:04.706037 3883 ts_tablet_manager.cc:1939] T d739e0c47a344284b637d47a0fc11f57 P b3d01ca389bf4a4fa503d849a23b71c6: Deleting consensus metadata
I20260619 18:45:04.707144 3370 catalog_manager.cc:4841] TS b3d01ca389bf4a4fa503d849a23b71c6 (127.21.143.67:39183): tablet d739e0c47a344284b637d47a0fc11f57 (table test-table [id=5a6323b44dfe4190b05d46e9ab6b429f]) successfully deleted
I20260619 18:45:04.707603 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 3713
I20260619 18:45:04.707818 3857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:04.707558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:42281 (local address 127.0.0.1:60706)
0619 18:45:04.707660 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:04.707726 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.65:42281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 18:45:04.708642 3857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:04.708374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:44607 (local address 127.0.0.1:48338)
0619 18:45:04.708509 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:04.708568 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.66:44607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 18:45:04.708863 3370 catalog_manager.cc:4584] TS 7e0eabab97f74bceaf95d7d08feea40c (127.21.143.66:44607): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d739e0c47a344284b637d47a0fc11f57: Network error: Client connection negotiation failed: client connection to 127.21.143.66:44607: connect: Connection refused (error 111)
I20260619 18:45:04.713660 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 3351
I20260619 18:45:04.719867 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:42911
--webserver_interface=127.21.143.126
--webserver_port=43643
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:42911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:45:04.829476 3885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:04.829739 3885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:04.829787 3885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:04.833807 3885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:45:04.833863 3885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:04.833881 3885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:04.833899 3885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:45:04.833913 3885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:45:04.839112 3885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:42911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:42911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=43643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:45:04.840301 3885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:04.841398 3885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:04.854346 3894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:04.854346 3892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:04.854347 3891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:04.854410 3885 server_base.cc:1034] running on GCE node
I20260619 18:45:04.855221 3885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:04.856469 3885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:04.857627 3885 hybrid_clock.cc:648] HybridClock initialized: now 1781894704857613 us; error 28 us; skew 500 ppm
Jun 19 18:45:04 dist-test-slave-mld5 krb5kdc[3338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894704, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:45:04.860396 3885 init.cc:377] Logged in from keytab as kudu/127.21.143.126@KRBTEST.COM (short username kudu)
I20260619 18:45:04.861284 3885 webserver.cc:460] Webserver started at http://127.21.143.126:43643/ using document root <none> and password file <none>
I20260619 18:45:04.861879 3885 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:04.861990 3885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:04.866039 3885 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:45:04.868273 3901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:04.869585 3885 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:45:04.869730 3885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "46b26682e2a74a828401a0e226e8ef74"
format_stamp: "Formatted at 2026-06-19 18:45:03 on dist-test-slave-mld5"
server_key: "2f215579542df4a63e9c2fd0b94af197"
server_key_iv: "6d6d23e3d59d005d35f09eb3e3075348"
server_key_version: "encryptionkey@0"
I20260619 18:45:04.870347 3885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:04.879837 3885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:04.882563 3885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:04.882767 3885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:04.889922 3885 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:42911
I20260619 18:45:04.890209 3953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:42911 every 8 connection(s)
I20260619 18:45:04.891088 3885 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:45:04.895463 3954 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:45:04.896052 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 3885
I20260619 18:45:04.896567 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.65:42281
--local_ip_for_outbound_sockets=127.21.143.65
--tserver_master_addrs=127.21.143.126:42911
--webserver_port=38075
--webserver_interface=127.21.143.65
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:45:04.897660 3954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74: Bootstrap starting.
I20260619 18:45:04.914893 3954 log.cc:824] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:04.923394 3954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:45:04.923753 3954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74: Bootstrap complete.
I20260619 18:45:04.927820 3954 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b26682e2a74a828401a0e226e8ef74" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42911 } }
I20260619 18:45:04.928215 3954 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46b26682e2a74a828401a0e226e8ef74, State: Initialized, Role: FOLLOWER
I20260619 18:45:04.928673 3954 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b26682e2a74a828401a0e226e8ef74" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42911 } }
I20260619 18:45:04.928848 3954 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:45:04.928934 3954 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:45:04.929054 3954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [term 1 FOLLOWER]: Advancing to term 2
I20260619 18:45:04.930912 3954 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b26682e2a74a828401a0e226e8ef74" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42911 } }
I20260619 18:45:04.931247 3954 leader_election.cc:302] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46b26682e2a74a828401a0e226e8ef74; no voters:
I20260619 18:45:04.931617 3954 leader_election.cc:288] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 18:45:04.931695 3958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 18:45:04.931936 3958 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [term 2 LEADER]: Becoming Leader. State: Replica: 46b26682e2a74a828401a0e226e8ef74, State: Running, Role: LEADER
I20260619 18:45:04.932278 3958 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b26682e2a74a828401a0e226e8ef74" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42911 } }
I20260619 18:45:04.932816 3954 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:45:04.933161 3958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46b26682e2a74a828401a0e226e8ef74. Latest consensus state: current_term: 2 leader_uuid: "46b26682e2a74a828401a0e226e8ef74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b26682e2a74a828401a0e226e8ef74" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42911 } } }
I20260619 18:45:04.933282 3958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [sys.catalog]: This master's current role is: LEADER
I20260619 18:45:04.933836 3959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "46b26682e2a74a828401a0e226e8ef74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b26682e2a74a828401a0e226e8ef74" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42911 } } }
I20260619 18:45:04.933991 3959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [sys.catalog]: This master's current role is: LEADER
I20260619 18:45:04.934595 3962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:45:04.939203 3962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:45:04.939951 3962 catalog_manager.cc:1259] Loaded cluster ID: a26e46602b274f61b30907f91272a69d
I20260619 18:45:04.940026 3962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:45:04.941567 3962 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 18:45:04.945914 3962 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 18:45:04.951112 3962 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:45:04.951886 3962 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74: Loaded TSK: 0
I20260619 18:45:04.953100 3962 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 18:45:05.013556 3956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:05.013824 3956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:05.013902 3956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:05.018006 3956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:05.018075 3956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:05.018169 3956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.65
I20260619 18:45:05.023607 3956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.65:42281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.143.65
--webserver_port=38075
--tserver_master_addrs=127.21.143.126:42911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.65
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:45:05.024861 3956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:05.025925 3956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:05.036955 3982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:05.036990 3984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:05.037151 3956 server_base.cc:1034] running on GCE node
W20260619 18:45:05.036962 3981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:05.037874 3956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:05.038681 3956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:05.039877 3956 hybrid_clock.cc:648] HybridClock initialized: now 1781894705039847 us; error 38 us; skew 500 ppm
Jun 19 18:45:05 dist-test-slave-mld5 krb5kdc[3338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894705, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:45:05.043445 3956 init.cc:377] Logged in from keytab as kudu/127.21.143.65@KRBTEST.COM (short username kudu)
I20260619 18:45:05.044591 3956 webserver.cc:460] Webserver started at http://127.21.143.65:38075/ using document root <none> and password file <none>
I20260619 18:45:05.045325 3956 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:05.045439 3956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:05.050617 3956 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:45:05.053256 3991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:05.054761 3956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:45:05.054881 3956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/wal
uuid: "9d8f9ac072db4db8aba2735fd7613aa0"
format_stamp: "Formatted at 2026-06-19 18:45:03 on dist-test-slave-mld5"
server_key: "bbb16c96d2ea28fd3f9622a935c0d0d3"
server_key_iv: "c5122a1d8301dc408d4cad3cbc084877"
server_key_version: "encryptionkey@0"
I20260619 18:45:05.055567 3956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:05.068627 3956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:05.071959 3956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:05.072175 3956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:05.072934 3956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:45:05.074213 3998 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 18:45:05.078469 3956 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 18:45:05.078532 3956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260619 18:45:05.078584 3956 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 18:45:05.080348 3956 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 18:45:05.080405 3956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.001s
I20260619 18:45:05.080672 3998 tablet_bootstrap.cc:492] T d739e0c47a344284b637d47a0fc11f57 P 9d8f9ac072db4db8aba2735fd7613aa0: Bootstrap starting.
I20260619 18:45:05.091357 3956 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.65:42281
I20260619 18:45:05.091434 4105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.65:42281 every 8 connection(s)
I20260619 18:45:05.092582 3956 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
I20260619 18:45:05.093598 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 3956
I20260619 18:45:05.094199 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.66:44607
--local_ip_for_outbound_sockets=127.21.143.66
--tserver_master_addrs=127.21.143.126:42911
--webserver_port=45415
--webserver_interface=127.21.143.66
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:45:05.102494 3998 log.cc:824] T d739e0c47a344284b637d47a0fc11f57 P 9d8f9ac072db4db8aba2735fd7613aa0: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:45:05 dist-test-slave-mld5 krb5kdc[3338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894705, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:45:05.110080 4108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:05.094580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:42695 (local address 127.21.143.126:42911)
0619 18:45:05.095045 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:05.095064 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:05.095795 (+ 731us) server_negotiation.cc:407] Connection header received
0619 18:45:05.097185 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:05.097200 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:05.097645 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:05.098052 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:05.099033 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.100083 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.101277 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.101906 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.102084 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:05.104870 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:05.104882 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:05.104895 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:05.104924 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:05.106722 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:05.107289 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:05.107295 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:05.107336 (+ 41us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:05.107469 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:05.107841 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:05.107845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:05.107846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:05.108224 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:05.108369 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:05.108904 (+ 535us) server_negotiation.cc:299] Negotiation successful
0619 18:45:05.109166 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":117,"threads_started":1}
I20260619 18:45:05.110118 4109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:05.094608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.65:42695)
0619 18:45:05.095355 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:05.095406 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:05.096611 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:05.098065 (+ 1454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:05.098072 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:05.098400 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:45:05.098856 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:05.098870 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.100298 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.100301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:05.101142 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:05.101150 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.102050 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.102054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:05.102250 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:05.102880 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:45:05.102896 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:45:05.104669 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:45:05.106865 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:05.106871 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:05.106894 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:05.107166 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:05.107593 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:05.107596 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:05.107598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:05.107740 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:05.108342 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:45:05.108348 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:45:05.108603 (+ 255us) client_negotiation.cc:769] Sending connection context
0619 18:45:05.108789 (+ 186us) client_negotiation.cc:240] Negotiation successful
0619 18:45:05.109154 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":132,"threads_started":1}
I20260619 18:45:05.111802 3998 tablet_bootstrap.cc:492] T d739e0c47a344284b637d47a0fc11f57 P 9d8f9ac072db4db8aba2735fd7613aa0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:45:05.112241 3998 tablet_bootstrap.cc:492] T d739e0c47a344284b637d47a0fc11f57 P 9d8f9ac072db4db8aba2735fd7613aa0: Bootstrap complete.
I20260619 18:45:05.112866 4106 heartbeater.cc:347] Connected to a master server at 127.21.143.126:42911
I20260619 18:45:05.113131 4106 heartbeater.cc:464] Registering TS with master...
I20260619 18:45:05.113651 4106 heartbeater.cc:511] Master 127.21.143.126:42911 requested a full tablet report, sending...
I20260619 18:45:05.113696 3998 ts_tablet_manager.cc:1397] T d739e0c47a344284b637d47a0fc11f57 P 9d8f9ac072db4db8aba2735fd7613aa0: Time spent bootstrapping tablet: real 0.033s user 0.026s sys 0.005s
I20260619 18:45:05.116395 3918 ts_manager.cc:194] Registered new tserver with Master: 9d8f9ac072db4db8aba2735fd7613aa0 (127.21.143.65:42281)
I20260619 18:45:05.116954 3998 raft_consensus.cc:348] T d739e0c47a344284b637d47a0fc11f57 P 9d8f9ac072db4db8aba2735fd7613aa0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d8f9ac072db4db8aba2735fd7613aa0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 42281 } } peers { permanent_uuid: "7e0eabab97f74bceaf95d7d08feea40c" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 44607 } } peers { permanent_uuid: "b3d01ca389bf4a4fa503d849a23b71c6" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 39183 } }
I20260619 18:45:05.117486 3998 raft_consensus.cc:729] T d739e0c47a344284b637d47a0fc11f57 P 9d8f9ac072db4db8aba2735fd7613aa0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d8f9ac072db4db8aba2735fd7613aa0, State: Initialized, Role: FOLLOWER
I20260619 18:45:05.117973 3998 consensus_queue.cc:260] T d739e0c47a344284b637d47a0fc11f57 P 9d8f9ac072db4db8aba2735fd7613aa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d8f9ac072db4db8aba2735fd7613aa0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 42281 } } peers { permanent_uuid: "7e0eabab97f74bceaf95d7d08feea40c" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 44607 } } peers { permanent_uuid: "b3d01ca389bf4a4fa503d849a23b71c6" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 39183 } }
I20260619 18:45:05.119043 3998 ts_tablet_manager.cc:1428] T d739e0c47a344284b637d47a0fc11f57 P 9d8f9ac072db4db8aba2735fd7613aa0: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 18:45:05.120136 3918 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.65@KRBTEST.COM'} at 127.21.143.65:42695
I20260619 18:45:05.122654 4106 heartbeater.cc:503] Master 127.21.143.126:42911 was elected leader, sending a full tablet report...
Jun 19 18:45:05 dist-test-slave-mld5 krb5kdc[3338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894704, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for kudu/127.21.143.65@KRBTEST.COM
I20260619 18:45:05.130301 4114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:05.119857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:42281 (local address 127.0.0.1:60712)
0619 18:45:05.120254 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:05.120309 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:05.120505 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:05.121038 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:05.121044 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:05.121495 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:45:05.121640 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:05.121649 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.122984 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.122989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:05.123767 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:05.123777 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.124459 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.124462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:05.124573 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:05.125129 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:45:05.125144 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:45:05.126620 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:45:05.128536 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:05.128545 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:05.128551 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:05.128782 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:05.129101 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:05.129104 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:05.129106 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:05.129193 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:05.129639 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:45:05.129645 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:45:05.129769 (+ 124us) client_negotiation.cc:769] Sending connection context
0619 18:45:05.129910 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 18:45:05.130109 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":103,"threads_started":1}
I20260619 18:45:05.130301 4113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:05.119518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60712 (local address 127.21.143.65:42281)
0619 18:45:05.119845 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:05.119857 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:05.120397 (+ 540us) server_negotiation.cc:407] Connection header received
0619 18:45:05.120692 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:05.120698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:05.120897 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:05.121051 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:05.121785 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.122859 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.123893 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.124323 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.124437 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:05.126757 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:05.126773 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:05.126782 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:05.126803 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:05.128403 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:05.128903 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:05.128909 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:05.128911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:05.128994 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:05.129285 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:05.129290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:05.129292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:05.129513 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:05.129664 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:05.129988 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 18:45:05.130144 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":104,"threads_started":1}
I20260619 18:45:05.131235 4040 tablet_service.cc:1514] Processing DeleteTablet for tablet d739e0c47a344284b637d47a0fc11f57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:45:04 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:60712
I20260619 18:45:05.132107 4115 tablet_replica.cc:330] stopping tablet replica
I20260619 18:45:05.132413 4115 raft_consensus.cc:2229] T d739e0c47a344284b637d47a0fc11f57 P 9d8f9ac072db4db8aba2735fd7613aa0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:45:05.132652 4115 raft_consensus.cc:2258] T d739e0c47a344284b637d47a0fc11f57 P 9d8f9ac072db4db8aba2735fd7613aa0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:45:05.133772 4115 ts_tablet_manager.cc:1905] T d739e0c47a344284b637d47a0fc11f57 P 9d8f9ac072db4db8aba2735fd7613aa0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:45:05.137962 4115 ts_tablet_manager.cc:1918] T d739e0c47a344284b637d47a0fc11f57 P 9d8f9ac072db4db8aba2735fd7613aa0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:45:05.138058 4115 log.cc:1196] T d739e0c47a344284b637d47a0fc11f57 P 9d8f9ac072db4db8aba2735fd7613aa0: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/wal/wals/d739e0c47a344284b637d47a0fc11f57
I20260619 18:45:05.138393 4115 ts_tablet_manager.cc:1939] T d739e0c47a344284b637d47a0fc11f57 P 9d8f9ac072db4db8aba2735fd7613aa0: Deleting consensus metadata
I20260619 18:45:05.140153 3902 catalog_manager.cc:4841] TS 9d8f9ac072db4db8aba2735fd7613aa0 (127.21.143.65:42281): tablet d739e0c47a344284b637d47a0fc11f57 (table test-table [id=5a6323b44dfe4190b05d46e9ab6b429f]) successfully deleted
W20260619 18:45:05.219129 4110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:05.219489 4110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:05.219575 4110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:05.223654 4110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:05.223734 4110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:05.223820 4110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.66
I20260619 18:45:05.228919 4110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.66:44607
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.143.66
--webserver_port=45415
--tserver_master_addrs=127.21.143.126:42911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.66
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:45:05.229930 4110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:05.230932 4110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:05.240688 4121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:05.240658 4123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:05.240787 4110 server_base.cc:1034] running on GCE node
W20260619 18:45:05.240653 4120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:05.241333 4110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:05.242007 4110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:05.243157 4110 hybrid_clock.cc:648] HybridClock initialized: now 1781894705243131 us; error 45 us; skew 500 ppm
Jun 19 18:45:05 dist-test-slave-mld5 krb5kdc[3338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894705, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:45:05.245882 4110 init.cc:377] Logged in from keytab as kudu/127.21.143.66@KRBTEST.COM (short username kudu)
I20260619 18:45:05.246718 4110 webserver.cc:460] Webserver started at http://127.21.143.66:45415/ using document root <none> and password file <none>
I20260619 18:45:05.247238 4110 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:05.247355 4110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:05.251457 4110 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260619 18:45:05.253645 4130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:05.254992 4110 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:45:05.255139 4110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/wal
uuid: "7e0eabab97f74bceaf95d7d08feea40c"
format_stamp: "Formatted at 2026-06-19 18:45:04 on dist-test-slave-mld5"
server_key: "67da6db9e2e65f66ac4ae634a9d3eb57"
server_key_iv: "fd5a7ee1d9a78ca5f054b06d94db421a"
server_key_version: "encryptionkey@0"
I20260619 18:45:05.255759 4110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:05.264691 4110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:05.267450 4110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:05.267637 4110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:05.268203 4110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:45:05.269232 4137 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260619 18:45:05.272986 4137 ts_tablet_manager.cc:1852] T d739e0c47a344284b637d47a0fc11f57 P 7e0eabab97f74bceaf95d7d08feea40c: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260619 18:45:05.273450 4137 ts_tablet_manager.cc:1905] T d739e0c47a344284b637d47a0fc11f57 P 7e0eabab97f74bceaf95d7d08feea40c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:45:05.277151 4137 ts_tablet_manager.cc:1918] T d739e0c47a344284b637d47a0fc11f57 P 7e0eabab97f74bceaf95d7d08feea40c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:45:05.277246 4137 log.cc:1196] T d739e0c47a344284b637d47a0fc11f57 P 7e0eabab97f74bceaf95d7d08feea40c: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/wal/wals/d739e0c47a344284b637d47a0fc11f57
I20260619 18:45:05.277500 4137 ts_tablet_manager.cc:1939] T d739e0c47a344284b637d47a0fc11f57 P 7e0eabab97f74bceaf95d7d08feea40c: Deleting consensus metadata
I20260619 18:45:05.277756 4110 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260619 18:45:05.277834 4110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260619 18:45:05.277921 4110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:45:05.277971 4110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:05.286782 4110 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.66:44607
I20260619 18:45:05.286818 4244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.66:44607 every 8 connection(s)
I20260619 18:45:05.287695 4110 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
I20260619 18:45:05.293735 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 4110
I20260619 18:45:05.294203 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.67:39183
--local_ip_for_outbound_sockets=127.21.143.67
--tserver_master_addrs=127.21.143.126:42911
--webserver_port=37647
--webserver_interface=127.21.143.67
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:45:05 dist-test-slave-mld5 krb5kdc[3338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894705, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:45:05.302721 4108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:05.289518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:57605 (local address 127.21.143.126:42911)
0619 18:45:05.289675 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:05.289680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:05.290587 (+ 907us) server_negotiation.cc:407] Connection header received
0619 18:45:05.291557 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:05.291560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:05.291608 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:05.291684 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:05.292845 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.293367 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.295029 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.295278 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.295743 (+ 465us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:05.298587 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:05.298594 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:05.298597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:05.298632 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:05.300238 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:05.300938 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:05.300941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:05.300942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:05.301020 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:05.301452 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:05.301455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:05.301456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:05.301640 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:05.301749 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:05.302465 (+ 716us) server_negotiation.cc:299] Negotiation successful
0619 18:45:05.302591 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:45:05.304124 4247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:05.289959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.66:57605)
0619 18:45:05.290437 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:05.290480 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:05.291293 (+ 813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:05.291850 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:05.291857 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:05.292186 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:45:05.292682 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:05.292696 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.293498 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.293502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:05.294890 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:05.294905 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.295657 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.295663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:05.295877 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:05.296510 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:45:05.296527 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:45:05.298359 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:45:05.300367 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:05.300376 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:05.300399 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:05.300826 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:05.301122 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:05.301128 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:05.301133 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:05.301351 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:05.301770 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:45:05.301780 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:45:05.302171 (+ 391us) client_negotiation.cc:769] Sending connection context
0619 18:45:05.302423 (+ 252us) client_negotiation.cc:240] Negotiation successful
0619 18:45:05.302838 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":130,"threads_started":1}
I20260619 18:45:05.305472 4245 heartbeater.cc:347] Connected to a master server at 127.21.143.126:42911
I20260619 18:45:05.305729 4245 heartbeater.cc:464] Registering TS with master...
I20260619 18:45:05.306226 4245 heartbeater.cc:511] Master 127.21.143.126:42911 requested a full tablet report, sending...
I20260619 18:45:05.307410 3918 ts_manager.cc:194] Registered new tserver with Master: 7e0eabab97f74bceaf95d7d08feea40c (127.21.143.66:44607)
I20260619 18:45:05.307997 3918 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.66@KRBTEST.COM'} at 127.21.143.66:57605
W20260619 18:45:05.409842 4248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:05.410367 4248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:05.410499 4248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:05.414613 4248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:05.414721 4248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:05.414813 4248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.67
I20260619 18:45:05.420293 4248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.67:39183
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.143.67
--webserver_port=37647
--tserver_master_addrs=127.21.143.126:42911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.67
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:45:05.421375 4248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:05.422400 4248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 18:45:05.431923 4248 server_base.cc:1034] running on GCE node
W20260619 18:45:05.431866 4254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:05.431833 4253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:05.431838 4256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:05.432596 4248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:05.433349 4248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:05.434502 4248 hybrid_clock.cc:648] HybridClock initialized: now 1781894705434479 us; error 42 us; skew 500 ppm
Jun 19 18:45:05 dist-test-slave-mld5 krb5kdc[3338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894705, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:45:05.437299 4248 init.cc:377] Logged in from keytab as kudu/127.21.143.67@KRBTEST.COM (short username kudu)
I20260619 18:45:05.438153 4248 webserver.cc:460] Webserver started at http://127.21.143.67:37647/ using document root <none> and password file <none>
I20260619 18:45:05.438669 4248 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:05.438740 4248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:05.442391 4248 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:45:05.444339 4263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:05.445514 4248 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:45:05.445605 4248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/wal
uuid: "b3d01ca389bf4a4fa503d849a23b71c6"
format_stamp: "Formatted at 2026-06-19 18:45:04 on dist-test-slave-mld5"
server_key: "51a67d461aecc2e01b9b9770862d2372"
server_key_iv: "3dd4c89f810e944bad8e5da56f0653bc"
server_key_version: "encryptionkey@0"
I20260619 18:45:05.446105 4248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:05.467016 4248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:05.469785 4248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:05.469935 4248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:05.470456 4248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:45:05.471221 4248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:45:05.471267 4248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:05.471364 4248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:45:05.471397 4248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:05.480640 4248 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.67:39183
I20260619 18:45:05.480667 4376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.67:39183 every 8 connection(s)
I20260619 18:45:05.481644 4248 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
Jun 19 18:45:05 dist-test-slave-mld5 krb5kdc[3338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894705, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:45:05.491601 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 4248
I20260619 18:45:05.495942 4108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:05.483515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:49089 (local address 127.21.143.126:42911)
0619 18:45:05.483664 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:05.483668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:05.484535 (+ 867us) server_negotiation.cc:407] Connection header received
0619 18:45:05.485932 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:05.485935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:05.485997 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:05.486106 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:05.487282 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.487867 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.489006 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.489243 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.489357 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:05.491914 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:05.491918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:05.491920 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:05.491944 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:05.493762 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:05.494351 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:05.494354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:05.494355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:05.494429 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:05.494779 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:05.494781 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:05.494782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:05.495037 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:05.495136 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:05.495667 (+ 531us) server_negotiation.cc:299] Negotiation successful
0619 18:45:05.495797 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 18:45:05.496804 4379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:05.483934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.67:49089)
0619 18:45:05.484381 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:05.484425 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:05.485716 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:05.486291 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:05.486298 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:05.486655 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:45:05.487126 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:05.487139 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.487994 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.487998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:05.488876 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:05.488885 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.489393 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.489397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:05.489566 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:05.490141 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:45:05.490156 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:45:05.491747 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:45:05.493889 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:05.493894 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:05.493906 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:05.494253 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:05.494516 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:05.494519 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:05.494521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:05.494694 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:05.495140 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:45:05.495145 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:45:05.495437 (+ 292us) client_negotiation.cc:769] Sending connection context
0619 18:45:05.495597 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 18:45:05.495901 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":107,"threads_started":1}
I20260619 18:45:05.498050 4377 heartbeater.cc:347] Connected to a master server at 127.21.143.126:42911
I20260619 18:45:05.498303 4377 heartbeater.cc:464] Registering TS with master...
I20260619 18:45:05.498895 4377 heartbeater.cc:511] Master 127.21.143.126:42911 requested a full tablet report, sending...
I20260619 18:45:05.500015 3918 ts_manager.cc:194] Registered new tserver with Master: b3d01ca389bf4a4fa503d849a23b71c6 (127.21.143.67:39183)
I20260619 18:45:05.500571 3918 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.67@KRBTEST.COM'} at 127.21.143.67:49089
I20260619 18:45:05.501044 4380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:05.492349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34554 (local address 127.21.143.126:42911)
0619 18:45:05.492886 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:05.492891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:05.492906 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:45:05.492943 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:05.492947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:05.492997 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:05.493075 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:05.494029 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.494768 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.495829 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.496154 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.496259 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:05.497763 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:05.497767 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:05.497769 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:05.497793 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:05.499357 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:05.499812 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:05.499815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:05.499817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:05.499899 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:05.500174 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:05.500179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:05.500181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:05.500359 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:05.500573 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:05.500775 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 18:45:05.500897 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":73,"threads_started":1}
I20260619 18:45:05.502710 22077 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:45:05.506845 4380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:05.503245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34562 (local address 127.21.143.126:42911)
0619 18:45:05.503394 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:05.503398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:05.503560 (+ 162us) server_negotiation.cc:407] Connection header received
0619 18:45:05.503738 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:05.503741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:05.503833 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:05.503935 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:45:05.504169 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.504664 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.505504 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.505740 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.505845 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:05.506149 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:45:05.506487 (+ 338us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:45:05.506616 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:05.506678 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 18:45:05.506729 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 18:45:05.507483 3918 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:34562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:45:05.510712 3918 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:45:05.518985 4040 tablet_service.cc:1467] Processing CreateTablet for tablet 2a7803e9963543818299d00c2680a9da (DEFAULT_TABLE table=test-table [id=afb27927b603415281e3591ab61b1fe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:45:05.519750 4040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a7803e9963543818299d00c2680a9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:05.522580 3998 tablet_bootstrap.cc:492] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0: Bootstrap starting.
I20260619 18:45:05.523046 4114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:05.517559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:44607 (local address 127.0.0.1:48348)
0619 18:45:05.517745 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:05.517761 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:05.517862 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:05.518750 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:05.518752 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:05.518767 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:05.518875 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:05.518882 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.520271 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.520276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:05.521445 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:05.521456 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.522567 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.522570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:05.522674 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:05.522729 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 18:45:05.522785 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 18:45:05.522880 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":20}
I20260619 18:45:05.523550 3998 tablet_bootstrap.cc:654] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:05.523768 4387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:05.517818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48348 (local address 127.21.143.66:44607)
0619 18:45:05.518227 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:05.518244 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:05.518273 (+ 29us) server_negotiation.cc:407] Connection header received
0619 18:45:05.518357 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:05.518362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:05.518609 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:05.518780 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:05.518987 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.520154 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.521585 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.522420 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.523239 (+ 819us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:05.523446 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:05.523501 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 18:45:05.523608 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":183,"threads_started":1}
I20260619 18:45:05.523782 4386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:05.517793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:39183 (local address 127.0.0.1:45278)
0619 18:45:05.518496 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:05.518509 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:05.518603 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:05.519048 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:05.519051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:05.519068 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:05.519158 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:05.519164 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.520737 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.520741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:05.521877 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:05.521886 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.523391 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.523395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:05.523483 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:05.523505 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 18:45:05.523583 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 18:45:05.523657 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":23,"thread_start_us":104,"threads_started":1}
I20260619 18:45:05.523929 4388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:05.518091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45278 (local address 127.21.143.67:39183)
0619 18:45:05.518470 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:05.518481 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:05.518604 (+ 123us) server_negotiation.cc:407] Connection header received
0619 18:45:05.518716 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:05.518721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:05.518910 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:05.519080 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:05.519256 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.520593 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.522025 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.523134 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.523243 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:05.523440 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:05.523643 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 18:45:05.523758 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":143,"threads_started":1}
I20260619 18:45:05.525173 3998 tablet_bootstrap.cc:492] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0: No bootstrap required, opened a new log
I20260619 18:45:05.525264 3998 ts_tablet_manager.cc:1397] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 18:45:05.525744 3998 raft_consensus.cc:348] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0eabab97f74bceaf95d7d08feea40c" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 44607 } } peers { permanent_uuid: "b3d01ca389bf4a4fa503d849a23b71c6" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 39183 } } peers { permanent_uuid: "9d8f9ac072db4db8aba2735fd7613aa0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 42281 } }
I20260619 18:45:05.525857 3998 raft_consensus.cc:374] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:05.525893 3998 raft_consensus.cc:729] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d8f9ac072db4db8aba2735fd7613aa0, State: Initialized, Role: FOLLOWER
I20260619 18:45:05.526098 3998 consensus_queue.cc:260] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0eabab97f74bceaf95d7d08feea40c" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 44607 } } peers { permanent_uuid: "b3d01ca389bf4a4fa503d849a23b71c6" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 39183 } } peers { permanent_uuid: "9d8f9ac072db4db8aba2735fd7613aa0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 42281 } }
I20260619 18:45:05.525794 4179 tablet_service.cc:1467] Processing CreateTablet for tablet 2a7803e9963543818299d00c2680a9da (DEFAULT_TABLE table=test-table [id=afb27927b603415281e3591ab61b1fe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:45:05.526381 3998 ts_tablet_manager.cc:1428] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:45:05.526971 4179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a7803e9963543818299d00c2680a9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:05.526975 4311 tablet_service.cc:1467] Processing CreateTablet for tablet 2a7803e9963543818299d00c2680a9da (DEFAULT_TABLE table=test-table [id=afb27927b603415281e3591ab61b1fe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:45:05.527933 4311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a7803e9963543818299d00c2680a9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:05.531870 4137 tablet_bootstrap.cc:492] T 2a7803e9963543818299d00c2680a9da P 7e0eabab97f74bceaf95d7d08feea40c: Bootstrap starting.
I20260619 18:45:05.534013 4137 tablet_bootstrap.cc:654] T 2a7803e9963543818299d00c2680a9da P 7e0eabab97f74bceaf95d7d08feea40c: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:05.534955 4137 log.cc:824] T 2a7803e9963543818299d00c2680a9da P 7e0eabab97f74bceaf95d7d08feea40c: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:05.535136 4390 tablet_bootstrap.cc:492] T 2a7803e9963543818299d00c2680a9da P b3d01ca389bf4a4fa503d849a23b71c6: Bootstrap starting.
I20260619 18:45:05.536736 4137 tablet_bootstrap.cc:492] T 2a7803e9963543818299d00c2680a9da P 7e0eabab97f74bceaf95d7d08feea40c: No bootstrap required, opened a new log
I20260619 18:45:05.536943 4137 ts_tablet_manager.cc:1397] T 2a7803e9963543818299d00c2680a9da P 7e0eabab97f74bceaf95d7d08feea40c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260619 18:45:05.537094 4390 tablet_bootstrap.cc:654] T 2a7803e9963543818299d00c2680a9da P b3d01ca389bf4a4fa503d849a23b71c6: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:05.538084 4390 log.cc:824] T 2a7803e9963543818299d00c2680a9da P b3d01ca389bf4a4fa503d849a23b71c6: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:05.539781 4390 tablet_bootstrap.cc:492] T 2a7803e9963543818299d00c2680a9da P b3d01ca389bf4a4fa503d849a23b71c6: No bootstrap required, opened a new log
I20260619 18:45:05.540005 4390 ts_tablet_manager.cc:1397] T 2a7803e9963543818299d00c2680a9da P b3d01ca389bf4a4fa503d849a23b71c6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260619 18:45:05.540027 4137 raft_consensus.cc:348] T 2a7803e9963543818299d00c2680a9da P 7e0eabab97f74bceaf95d7d08feea40c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0eabab97f74bceaf95d7d08feea40c" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 44607 } } peers { permanent_uuid: "b3d01ca389bf4a4fa503d849a23b71c6" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 39183 } } peers { permanent_uuid: "9d8f9ac072db4db8aba2735fd7613aa0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 42281 } }
I20260619 18:45:05.540306 4137 raft_consensus.cc:374] T 2a7803e9963543818299d00c2680a9da P 7e0eabab97f74bceaf95d7d08feea40c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:05.540412 4137 raft_consensus.cc:729] T 2a7803e9963543818299d00c2680a9da P 7e0eabab97f74bceaf95d7d08feea40c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e0eabab97f74bceaf95d7d08feea40c, State: Initialized, Role: FOLLOWER
I20260619 18:45:05.540866 4137 consensus_queue.cc:260] T 2a7803e9963543818299d00c2680a9da P 7e0eabab97f74bceaf95d7d08feea40c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0eabab97f74bceaf95d7d08feea40c" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 44607 } } peers { permanent_uuid: "b3d01ca389bf4a4fa503d849a23b71c6" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 39183 } } peers { permanent_uuid: "9d8f9ac072db4db8aba2735fd7613aa0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 42281 } }
I20260619 18:45:05.541527 4245 heartbeater.cc:503] Master 127.21.143.126:42911 was elected leader, sending a full tablet report...
I20260619 18:45:05.541841 4137 ts_tablet_manager.cc:1428] T 2a7803e9963543818299d00c2680a9da P 7e0eabab97f74bceaf95d7d08feea40c: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260619 18:45:05.543503 4390 raft_consensus.cc:348] T 2a7803e9963543818299d00c2680a9da P b3d01ca389bf4a4fa503d849a23b71c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0eabab97f74bceaf95d7d08feea40c" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 44607 } } peers { permanent_uuid: "b3d01ca389bf4a4fa503d849a23b71c6" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 39183 } } peers { permanent_uuid: "9d8f9ac072db4db8aba2735fd7613aa0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 42281 } }
I20260619 18:45:05.543740 4390 raft_consensus.cc:374] T 2a7803e9963543818299d00c2680a9da P b3d01ca389bf4a4fa503d849a23b71c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:05.543795 4390 raft_consensus.cc:729] T 2a7803e9963543818299d00c2680a9da P b3d01ca389bf4a4fa503d849a23b71c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3d01ca389bf4a4fa503d849a23b71c6, State: Initialized, Role: FOLLOWER
I20260619 18:45:05.544216 4390 consensus_queue.cc:260] T 2a7803e9963543818299d00c2680a9da P b3d01ca389bf4a4fa503d849a23b71c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0eabab97f74bceaf95d7d08feea40c" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 44607 } } peers { permanent_uuid: "b3d01ca389bf4a4fa503d849a23b71c6" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 39183 } } peers { permanent_uuid: "9d8f9ac072db4db8aba2735fd7613aa0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 42281 } }
I20260619 18:45:05.544847 4377 heartbeater.cc:503] Master 127.21.143.126:42911 was elected leader, sending a full tablet report...
I20260619 18:45:05.545060 4390 ts_tablet_manager.cc:1428] T 2a7803e9963543818299d00c2680a9da P b3d01ca389bf4a4fa503d849a23b71c6: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
W20260619 18:45:05.593827 4107 tablet.cc:2369] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:45:05.649353 4393 raft_consensus.cc:482] T 2a7803e9963543818299d00c2680a9da P 7e0eabab97f74bceaf95d7d08feea40c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:45:05.649554 4393 raft_consensus.cc:504] T 2a7803e9963543818299d00c2680a9da P 7e0eabab97f74bceaf95d7d08feea40c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0eabab97f74bceaf95d7d08feea40c" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 44607 } } peers { permanent_uuid: "b3d01ca389bf4a4fa503d849a23b71c6" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 39183 } } peers { permanent_uuid: "9d8f9ac072db4db8aba2735fd7613aa0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 42281 } }
I20260619 18:45:05.650753 4393 leader_election.cc:288] T 2a7803e9963543818299d00c2680a9da P 7e0eabab97f74bceaf95d7d08feea40c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3d01ca389bf4a4fa503d849a23b71c6 (127.21.143.67:39183), 9d8f9ac072db4db8aba2735fd7613aa0 (127.21.143.65:42281)
I20260619 18:45:05.655578 4388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:05.651065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:55079 (local address 127.21.143.67:39183)
0619 18:45:05.651427 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:05.651433 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:05.651449 (+ 16us) server_negotiation.cc:407] Connection header received
0619 18:45:05.651508 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:05.651511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:05.651560 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:05.651649 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:05.651939 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.652708 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.654353 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.654911 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.655036 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:05.655101 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:05.655377 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 18:45:05.655446 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273}
I20260619 18:45:05.655592 4396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:05.651234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:57701 (local address 127.21.143.65:42281)
0619 18:45:05.651489 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:05.651493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:05.651546 (+ 53us) server_negotiation.cc:407] Connection header received
0619 18:45:05.651677 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:05.651680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:05.651741 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:05.651839 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:05.652121 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.652847 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.654353 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.654989 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.655088 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:05.655219 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:05.655376 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 18:45:05.655446 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":95,"threads_started":1}
I20260619 18:45:05.655654 4247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:05.650955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:39183 (local address 127.21.143.66:55079)
0619 18:45:05.651087 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:05.651103 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:05.651211 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:05.651679 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:05.651683 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:05.651697 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:05.651799 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:05.651803 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.652811 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.652814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:05.654222 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:05.654232 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.655052 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.655059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:05.655179 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:05.655270 (+ 91us) client_negotiation.cc:769] Sending connection context
0619 18:45:05.655364 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 18:45:05.655482 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":149}
I20260619 18:45:05.655663 4395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:05.651031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:42281 (local address 127.21.143.66:57701)
0619 18:45:05.651485 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:05.651496 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:05.651575 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:05.651878 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:05.651881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:05.651910 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:05.652008 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:05.652012 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.652960 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.652962 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:05.654220 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:05.654232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.655086 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.655089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:05.655182 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:05.655273 (+ 91us) client_negotiation.cc:769] Sending connection context
0619 18:45:05.655364 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 18:45:05.655482 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":80,"thread_start_us":143,"threads_started":1}
I20260619 18:45:05.656378 4331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a7803e9963543818299d00c2680a9da" candidate_uuid: "7e0eabab97f74bceaf95d7d08feea40c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d01ca389bf4a4fa503d849a23b71c6" is_pre_election: true
I20260619 18:45:05.656567 4060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a7803e9963543818299d00c2680a9da" candidate_uuid: "7e0eabab97f74bceaf95d7d08feea40c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d8f9ac072db4db8aba2735fd7613aa0" is_pre_election: true
I20260619 18:45:05.656754 4331 raft_consensus.cc:2454] T 2a7803e9963543818299d00c2680a9da P b3d01ca389bf4a4fa503d849a23b71c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e0eabab97f74bceaf95d7d08feea40c in term 0.
I20260619 18:45:05.656797 4060 raft_consensus.cc:2454] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e0eabab97f74bceaf95d7d08feea40c in term 0.
I20260619 18:45:05.657276 4134 leader_election.cc:302] T 2a7803e9963543818299d00c2680a9da P 7e0eabab97f74bceaf95d7d08feea40c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e0eabab97f74bceaf95d7d08feea40c, b3d01ca389bf4a4fa503d849a23b71c6; no voters:
I20260619 18:45:05.657591 4393 raft_consensus.cc:2781] T 2a7803e9963543818299d00c2680a9da P 7e0eabab97f74bceaf95d7d08feea40c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:45:05.657693 4393 raft_consensus.cc:482] T 2a7803e9963543818299d00c2680a9da P 7e0eabab97f74bceaf95d7d08feea40c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:45:05.657754 4393 raft_consensus.cc:3037] T 2a7803e9963543818299d00c2680a9da P 7e0eabab97f74bceaf95d7d08feea40c [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:45:05.658924 4393 raft_consensus.cc:504] T 2a7803e9963543818299d00c2680a9da P 7e0eabab97f74bceaf95d7d08feea40c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0eabab97f74bceaf95d7d08feea40c" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 44607 } } peers { permanent_uuid: "b3d01ca389bf4a4fa503d849a23b71c6" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 39183 } } peers { permanent_uuid: "9d8f9ac072db4db8aba2735fd7613aa0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 42281 } }
I20260619 18:45:05.659377 4393 leader_election.cc:288] T 2a7803e9963543818299d00c2680a9da P 7e0eabab97f74bceaf95d7d08feea40c [CANDIDATE]: Term 1 election: Requested vote from peers b3d01ca389bf4a4fa503d849a23b71c6 (127.21.143.67:39183), 9d8f9ac072db4db8aba2735fd7613aa0 (127.21.143.65:42281)
I20260619 18:45:05.659615 4331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a7803e9963543818299d00c2680a9da" candidate_uuid: "7e0eabab97f74bceaf95d7d08feea40c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d01ca389bf4a4fa503d849a23b71c6"
I20260619 18:45:05.659740 4331 raft_consensus.cc:3037] T 2a7803e9963543818299d00c2680a9da P b3d01ca389bf4a4fa503d849a23b71c6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:45:05.659745 4060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a7803e9963543818299d00c2680a9da" candidate_uuid: "7e0eabab97f74bceaf95d7d08feea40c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d8f9ac072db4db8aba2735fd7613aa0"
I20260619 18:45:05.659868 4060 raft_consensus.cc:3037] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:45:05.660835 4331 raft_consensus.cc:2454] T 2a7803e9963543818299d00c2680a9da P b3d01ca389bf4a4fa503d849a23b71c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e0eabab97f74bceaf95d7d08feea40c in term 1.
I20260619 18:45:05.661089 4060 raft_consensus.cc:2454] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e0eabab97f74bceaf95d7d08feea40c in term 1.
I20260619 18:45:05.661113 4134 leader_election.cc:302] T 2a7803e9963543818299d00c2680a9da P 7e0eabab97f74bceaf95d7d08feea40c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e0eabab97f74bceaf95d7d08feea40c, b3d01ca389bf4a4fa503d849a23b71c6; no voters:
I20260619 18:45:05.661268 4393 raft_consensus.cc:2781] T 2a7803e9963543818299d00c2680a9da P 7e0eabab97f74bceaf95d7d08feea40c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:45:05.661522 4393 raft_consensus.cc:686] T 2a7803e9963543818299d00c2680a9da P 7e0eabab97f74bceaf95d7d08feea40c [term 1 LEADER]: Becoming Leader. State: Replica: 7e0eabab97f74bceaf95d7d08feea40c, State: Running, Role: LEADER
I20260619 18:45:05.661870 4393 consensus_queue.cc:237] T 2a7803e9963543818299d00c2680a9da P 7e0eabab97f74bceaf95d7d08feea40c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0eabab97f74bceaf95d7d08feea40c" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 44607 } } peers { permanent_uuid: "b3d01ca389bf4a4fa503d849a23b71c6" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 39183 } } peers { permanent_uuid: "9d8f9ac072db4db8aba2735fd7613aa0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 42281 } }
I20260619 18:45:05.665059 3917 catalog_manager.cc:5495] T 2a7803e9963543818299d00c2680a9da P 7e0eabab97f74bceaf95d7d08feea40c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e0eabab97f74bceaf95d7d08feea40c (127.21.143.66). New cstate: current_term: 1 leader_uuid: "7e0eabab97f74bceaf95d7d08feea40c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0eabab97f74bceaf95d7d08feea40c" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 44607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3d01ca389bf4a4fa503d849a23b71c6" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d8f9ac072db4db8aba2735fd7613aa0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 42281 } health_report { overall_health: UNKNOWN } } }
I20260619 18:45:05.675679 4387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:05.672147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48352 (local address 127.21.143.66:44607)
0619 18:45:05.672270 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:05.672274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:05.672289 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:45:05.672466 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:05.672468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:05.672521 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:05.672607 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:45:05.672794 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.673279 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.674217 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:05.674453 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:05.674535 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:05.674824 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:45:05.675285 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:45:05.675445 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:05.675499 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 18:45:05.675550 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 18:45:05.681229 4060 raft_consensus.cc:1261] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [term 1 FOLLOWER]: Refusing update from remote peer 7e0eabab97f74bceaf95d7d08feea40c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:45:05.681232 4331 raft_consensus.cc:1261] T 2a7803e9963543818299d00c2680a9da P b3d01ca389bf4a4fa503d849a23b71c6 [term 1 FOLLOWER]: Refusing update from remote peer 7e0eabab97f74bceaf95d7d08feea40c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:45:05.681943 4397 consensus_queue.cc:1035] T 2a7803e9963543818299d00c2680a9da P 7e0eabab97f74bceaf95d7d08feea40c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d01ca389bf4a4fa503d849a23b71c6" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:45:05.682103 4393 consensus_queue.cc:1035] T 2a7803e9963543818299d00c2680a9da P 7e0eabab97f74bceaf95d7d08feea40c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d8f9ac072db4db8aba2735fd7613aa0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 42281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:45:05.688851 4400 mvcc.cc:205] Tried to move back new op lower bound from 7298640714464014336 to 7298640714394247168. Current Snapshot: MvccSnapshot[applied={T|T < 7298640714464014336}]
I20260619 18:45:05.692140 4401 mvcc.cc:205] Tried to move back new op lower bound from 7298640714464014336 to 7298640714394247168. Current Snapshot: MvccSnapshot[applied={T|T < 7298640714464014336 or (T in {7298640714464014336})}]
I20260619 18:45:05.696482 3918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:34562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:45:05.696633 3918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:34562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:45:05.698990 3918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74: Sending DeleteTablet for 3 replicas of tablet 2a7803e9963543818299d00c2680a9da
I20260619 18:45:05.699797 4040 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a7803e9963543818299d00c2680a9da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:45:05 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:60712
I20260619 18:45:05.699823 4179 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a7803e9963543818299d00c2680a9da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:45:05 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:48348
I20260619 18:45:05.699839 4311 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a7803e9963543818299d00c2680a9da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:45:05 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:45278
I20260619 18:45:05.700289 4408 tablet_replica.cc:330] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0: stopping tablet replica
I20260619 18:45:05.700414 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 3956
I20260619 18:45:05.700449 4408 raft_consensus.cc:2229] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:45:05.700524 4409 tablet_replica.cc:330] T 2a7803e9963543818299d00c2680a9da P 7e0eabab97f74bceaf95d7d08feea40c: stopping tablet replica
I20260619 18:45:05.700784 4409 raft_consensus.cc:2229] T 2a7803e9963543818299d00c2680a9da P 7e0eabab97f74bceaf95d7d08feea40c [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:45:05.701149 4409 raft_consensus.cc:2258] T 2a7803e9963543818299d00c2680a9da P 7e0eabab97f74bceaf95d7d08feea40c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:45:05.702487 4409 ts_tablet_manager.cc:1905] T 2a7803e9963543818299d00c2680a9da P 7e0eabab97f74bceaf95d7d08feea40c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:45:05.705511 4409 ts_tablet_manager.cc:1918] T 2a7803e9963543818299d00c2680a9da P 7e0eabab97f74bceaf95d7d08feea40c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:45:05.705611 4409 log.cc:1196] T 2a7803e9963543818299d00c2680a9da P 7e0eabab97f74bceaf95d7d08feea40c: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/wal/wals/2a7803e9963543818299d00c2680a9da
I20260619 18:45:05.705891 4409 ts_tablet_manager.cc:1939] T 2a7803e9963543818299d00c2680a9da P 7e0eabab97f74bceaf95d7d08feea40c: Deleting consensus metadata
I20260619 18:45:05.707223 4410 tablet_replica.cc:330] T 2a7803e9963543818299d00c2680a9da P b3d01ca389bf4a4fa503d849a23b71c6: stopping tablet replica
I20260619 18:45:05.707221 3905 catalog_manager.cc:4841] TS 7e0eabab97f74bceaf95d7d08feea40c (127.21.143.66:44607): tablet 2a7803e9963543818299d00c2680a9da (table test-table [id=afb27927b603415281e3591ab61b1fe9]) successfully deleted
I20260619 18:45:05.707687 4410 raft_consensus.cc:2229] T 2a7803e9963543818299d00c2680a9da P b3d01ca389bf4a4fa503d849a23b71c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:45:05.708045 4410 raft_consensus.cc:2258] T 2a7803e9963543818299d00c2680a9da P b3d01ca389bf4a4fa503d849a23b71c6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 18:45:05.709266 3902 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.143.65:42281 (error 108)
I20260619 18:45:05.709589 4410 ts_tablet_manager.cc:1905] T 2a7803e9963543818299d00c2680a9da P b3d01ca389bf4a4fa503d849a23b71c6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 18:45:05.710697 4386 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:45:05.709988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:42281 (local address 127.0.0.1:60714)
0619 18:45:05.710117 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:05.710135 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:05.710223 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:05.710567 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.65:42281: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 18:45:05.710840 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 4110
W20260619 18:45:05.710999 3902 catalog_manager.cc:4584] TS 9d8f9ac072db4db8aba2735fd7613aa0 (127.21.143.65:42281): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a7803e9963543818299d00c2680a9da: Network error: Client connection negotiation failed: client connection to 127.21.143.65:42281: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 18:45:05.712267 4410 ts_tablet_manager.cc:1918] T 2a7803e9963543818299d00c2680a9da P b3d01ca389bf4a4fa503d849a23b71c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:45:05.712376 4410 log.cc:1196] T 2a7803e9963543818299d00c2680a9da P b3d01ca389bf4a4fa503d849a23b71c6: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/wal/wals/2a7803e9963543818299d00c2680a9da
I20260619 18:45:05.712749 4410 ts_tablet_manager.cc:1939] T 2a7803e9963543818299d00c2680a9da P b3d01ca389bf4a4fa503d849a23b71c6: Deleting consensus metadata
I20260619 18:45:05.714037 3905 catalog_manager.cc:4841] TS b3d01ca389bf4a4fa503d849a23b71c6 (127.21.143.67:39183): tablet 2a7803e9963543818299d00c2680a9da (table test-table [id=afb27927b603415281e3591ab61b1fe9]) successfully deleted
I20260619 18:45:05.717743 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 4248
I20260619 18:45:05.723244 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 3885
I20260619 18:45:05.729640 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:42911
--webserver_interface=127.21.143.126
--webserver_port=43643
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:42911
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:45:05.837230 4412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:05.837502 4412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:05.837543 4412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:05.841420 4412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:45:05.841467 4412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:05.841486 4412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:05.841502 4412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:45:05.841517 4412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:45:05.846668 4412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:42911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:42911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=43643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:45:05.847630 4412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:05.848554 4412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:05.856930 4417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:05.857091 4412 server_base.cc:1034] running on GCE node
W20260619 18:45:05.856974 4418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:05.856993 4420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:05.857784 4412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:05.858902 4412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:05.860129 4412 hybrid_clock.cc:648] HybridClock initialized: now 1781894705860102 us; error 38 us; skew 500 ppm
Jun 19 18:45:05 dist-test-slave-mld5 krb5kdc[3338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894705, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:45:05.863003 4412 init.cc:377] Logged in from keytab as kudu/127.21.143.126@KRBTEST.COM (short username kudu)
I20260619 18:45:05.863988 4412 webserver.cc:460] Webserver started at http://127.21.143.126:43643/ using document root <none> and password file <none>
I20260619 18:45:05.864754 4412 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:05.864938 4412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:05.868961 4412 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 18:45:05.871186 4427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:05.872479 4412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:45:05.872640 4412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "46b26682e2a74a828401a0e226e8ef74"
format_stamp: "Formatted at 2026-06-19 18:45:03 on dist-test-slave-mld5"
server_key: "2f215579542df4a63e9c2fd0b94af197"
server_key_iv: "6d6d23e3d59d005d35f09eb3e3075348"
server_key_version: "encryptionkey@0"
I20260619 18:45:05.873224 4412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:05.890760 4412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:05.893776 4412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:05.894004 4412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:05.901404 4412 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:42911
I20260619 18:45:05.901805 4479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:42911 every 8 connection(s)
I20260619 18:45:05.902513 4412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:45:05.905691 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 4412
I20260619 18:45:05.906104 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.65:42281
--local_ip_for_outbound_sockets=127.21.143.65
--tserver_master_addrs=127.21.143.126:42911
--webserver_port=38075
--webserver_interface=127.21.143.65
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:45:05.906709 4480 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:45:05.908355 4480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74: Bootstrap starting.
I20260619 18:45:05.933182 4480 log.cc:824] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:05.947402 4480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:45:05.947860 4480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74: Bootstrap complete.
I20260619 18:45:05.953464 4480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b26682e2a74a828401a0e226e8ef74" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42911 } }
I20260619 18:45:05.953979 4480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46b26682e2a74a828401a0e226e8ef74, State: Initialized, Role: FOLLOWER
I20260619 18:45:05.954531 4480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b26682e2a74a828401a0e226e8ef74" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42911 } }
I20260619 18:45:05.954669 4480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:45:05.954736 4480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:45:05.954829 4480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [term 2 FOLLOWER]: Advancing to term 3
I20260619 18:45:05.957293 4480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b26682e2a74a828401a0e226e8ef74" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42911 } }
I20260619 18:45:05.957648 4480 leader_election.cc:302] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46b26682e2a74a828401a0e226e8ef74; no voters:
I20260619 18:45:05.958017 4480 leader_election.cc:288] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 18:45:05.958142 4484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 18:45:05.958401 4484 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [term 3 LEADER]: Becoming Leader. State: Replica: 46b26682e2a74a828401a0e226e8ef74, State: Running, Role: LEADER
I20260619 18:45:05.958746 4484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b26682e2a74a828401a0e226e8ef74" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42911 } }
I20260619 18:45:05.959226 4480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:45:05.960084 4485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "46b26682e2a74a828401a0e226e8ef74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b26682e2a74a828401a0e226e8ef74" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42911 } } }
I20260619 18:45:05.960194 4485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [sys.catalog]: This master's current role is: LEADER
I20260619 18:45:05.960575 4486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46b26682e2a74a828401a0e226e8ef74. Latest consensus state: current_term: 3 leader_uuid: "46b26682e2a74a828401a0e226e8ef74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b26682e2a74a828401a0e226e8ef74" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42911 } } }
I20260619 18:45:05.960686 4486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46b26682e2a74a828401a0e226e8ef74 [sys.catalog]: This master's current role is: LEADER
I20260619 18:45:05.960963 4493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:45:05.966076 4493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:45:05.966706 4493 catalog_manager.cc:1259] Loaded cluster ID: a26e46602b274f61b30907f91272a69d
I20260619 18:45:05.966768 4493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 18:45:05.967792 4493 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f47458989ad google::LogMessage::Fail() at ??:0
@ 0x7f474589b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f47458984de google::LogMessage::Flush() at ??:0
@ 0x7f4745899a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f47501d7fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f47501d8d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f47501d5579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f4750219640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4747aea3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4747aeac83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f4747aec3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4747adb512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f474911f6db start_thread at ??:0
@ 0x7f474439661f clone at ??:0
W20260619 18:45:06.065626 4482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:06.066051 4482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:06.066125 4482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:06.072520 4482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:06.072618 4482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:06.072710 4482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.65
I20260619 18:45:06.080850 4482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.65:42281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.143.65
--webserver_port=38075
--tserver_master_addrs=127.21.143.126:42911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.65
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:45:06.082304 4482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:06.083710 4482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 18:45:06.095561 4482 server_base.cc:1034] running on GCE node
W20260619 18:45:06.095499 4521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:06.095746 4524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:06.095499 4522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:06.096179 4482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:06.096887 4482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:06.098071 4482 hybrid_clock.cc:648] HybridClock initialized: now 1781894706098037 us; error 52 us; skew 500 ppm
Jun 19 18:45:06 dist-test-slave-mld5 krb5kdc[3338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894706, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:45:06.101177 4482 init.cc:377] Logged in from keytab as kudu/127.21.143.65@KRBTEST.COM (short username kudu)
I20260619 18:45:06.102178 4482 webserver.cc:460] Webserver started at http://127.21.143.65:38075/ using document root <none> and password file <none>
I20260619 18:45:06.102788 4482 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:06.102864 4482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:06.107105 4482 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:45:06.109580 4531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:06.110971 4482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:45:06.111131 4482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/wal
uuid: "9d8f9ac072db4db8aba2735fd7613aa0"
format_stamp: "Formatted at 2026-06-19 18:45:03 on dist-test-slave-mld5"
server_key: "bbb16c96d2ea28fd3f9622a935c0d0d3"
server_key_iv: "c5122a1d8301dc408d4cad3cbc084877"
server_key_version: "encryptionkey@0"
I20260619 18:45:06.111785 4482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:06.122640 4482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:06.125917 4482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:06.126122 4482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:06.126715 4482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:45:06.127893 4538 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 18:45:06.133390 4482 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 18:45:06.133487 4482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260619 18:45:06.133563 4482 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 18:45:06.135217 4482 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 18:45:06.135288 4482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 18:45:06.135651 4538 tablet_bootstrap.cc:492] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0: Bootstrap starting.
I20260619 18:45:06.145896 4482 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.65:42281
I20260619 18:45:06.146147 4645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.65:42281 every 8 connection(s)
I20260619 18:45:06.146922 4482 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
I20260619 18:45:06.153988 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 4482
I20260619 18:45:06.154408 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.66:44607
--local_ip_for_outbound_sockets=127.21.143.66
--tserver_master_addrs=127.21.143.126:42911
--webserver_port=45415
--webserver_interface=127.21.143.66
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:45:06.157049 4538 log.cc:824] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:06.166095 4538 tablet_bootstrap.cc:492] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:45:06.166561 4538 tablet_bootstrap.cc:492] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0: Bootstrap complete.
I20260619 18:45:06.168037 4538 ts_tablet_manager.cc:1397] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0: Time spent bootstrapping tablet: real 0.033s user 0.020s sys 0.011s
I20260619 18:45:06.171420 4538 raft_consensus.cc:348] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0eabab97f74bceaf95d7d08feea40c" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 44607 } } peers { permanent_uuid: "b3d01ca389bf4a4fa503d849a23b71c6" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 39183 } } peers { permanent_uuid: "9d8f9ac072db4db8aba2735fd7613aa0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 42281 } }
I20260619 18:45:06.172037 4538 raft_consensus.cc:729] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d8f9ac072db4db8aba2735fd7613aa0, State: Initialized, Role: FOLLOWER
I20260619 18:45:06.172640 4538 consensus_queue.cc:260] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0eabab97f74bceaf95d7d08feea40c" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 44607 } } peers { permanent_uuid: "b3d01ca389bf4a4fa503d849a23b71c6" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 39183 } } peers { permanent_uuid: "9d8f9ac072db4db8aba2735fd7613aa0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 42281 } }
I20260619 18:45:06.173700 4538 ts_tablet_manager.cc:1428] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
W20260619 18:45:06.219817 4648 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:45:06.149045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.65:47623)
0619 18:45:06.149579 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:06.149633 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:06.150440 (+ 807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:06.218759 (+ 68319us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":190,"threads_started":1}
W20260619 18:45:06.220369 4646 heartbeater.cc:644] Failed to heartbeat to 127.21.143.126:42911 (0 consecutive failures): Network error: Failed to ping master at 127.21.143.126:42911: Client connection negotiation failed: client connection to 127.21.143.126:42911: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 18:45:06.221159 4648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:06.220868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.65:59067)
0619 18:45:06.221010 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:06.221091 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 18:45:06.221830 4648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:06.221618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.65:34989)
0619 18:45:06.221724 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:06.221774 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 18:45:06.222026 4646 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 18:45:06.272312 4649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:06.272589 4649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:06.272634 4649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:06.276867 4649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:06.276957 4649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:06.277047 4649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.66
I20260619 18:45:06.282642 4649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.66:44607
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.143.66
--webserver_port=45415
--tserver_master_addrs=127.21.143.126:42911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.66
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:45:06.283839 4649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:06.284948 4649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:06.294695 4657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:06.294682 4656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:06.294682 4659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:06.294993 4649 server_base.cc:1034] running on GCE node
I20260619 18:45:06.295614 4649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:06.296285 4649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:06.297457 4649 hybrid_clock.cc:648] HybridClock initialized: now 1781894706297428 us; error 51 us; skew 500 ppm
Jun 19 18:45:06 dist-test-slave-mld5 krb5kdc[3338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894706, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:45:06.300400 4649 init.cc:377] Logged in from keytab as kudu/127.21.143.66@KRBTEST.COM (short username kudu)
I20260619 18:45:06.301328 4649 webserver.cc:460] Webserver started at http://127.21.143.66:45415/ using document root <none> and password file <none>
I20260619 18:45:06.301968 4649 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:06.302065 4649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:06.306201 4649 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:45:06.308444 4666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:06.309743 4649 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:45:06.309897 4649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/wal
uuid: "7e0eabab97f74bceaf95d7d08feea40c"
format_stamp: "Formatted at 2026-06-19 18:45:04 on dist-test-slave-mld5"
server_key: "67da6db9e2e65f66ac4ae634a9d3eb57"
server_key_iv: "fd5a7ee1d9a78ca5f054b06d94db421a"
server_key_version: "encryptionkey@0"
I20260619 18:45:06.310477 4649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:06.328974 4649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:06.331969 4649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:06.332185 4649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:06.332801 4649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:45:06.333658 4649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:45:06.333730 4649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:06.333801 4649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:45:06.333844 4649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:06.342779 4649 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.66:44607
I20260619 18:45:06.342810 4779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.66:44607 every 8 connection(s)
I20260619 18:45:06.343865 4649 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
I20260619 18:45:06.347765 4782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:06.345978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.66:53079)
0619 18:45:06.346412 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:06.346532 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":105,"threads_started":1}
W20260619 18:45:06.348223 4780 heartbeater.cc:644] Failed to heartbeat to 127.21.143.126:42911 (0 consecutive failures): Network error: Failed to ping master at 127.21.143.126:42911: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
I20260619 18:45:06.348846 4782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:06.348635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.66:55767)
0619 18:45:06.348741 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:06.348794 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 18:45:06.349426 4782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:06.349214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.66:33131)
0619 18:45:06.349333 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:06.349379 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 18:45:06.349573 4780 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 18:45:06.351856 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 4649
I20260619 18:45:06.352252 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.67:39183
--local_ip_for_outbound_sockets=127.21.143.67
--tserver_master_addrs=127.21.143.126:42911
--webserver_port=37647
--webserver_interface=127.21.143.67
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:45:06.464047 4783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:06.464316 4783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:06.464360 4783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:06.468375 4783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:06.468436 4783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:06.468534 4783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.67
I20260619 18:45:06.473759 4783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.67:39183
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.143.67
--webserver_port=37647
--tserver_master_addrs=127.21.143.126:42911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.67
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:45:06.474826 4783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:06.475952 4783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 18:45:06.485497 4783 server_base.cc:1034] running on GCE node
W20260619 18:45:06.485433 4789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:06.485422 4788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:06.485427 4791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:06.486059 4783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:06.486656 4783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:06.487829 4783 hybrid_clock.cc:648] HybridClock initialized: now 1781894706487798 us; error 39 us; skew 500 ppm
Jun 19 18:45:06 dist-test-slave-mld5 krb5kdc[3338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894706, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:45:06.490468 4783 init.cc:377] Logged in from keytab as kudu/127.21.143.67@KRBTEST.COM (short username kudu)
I20260619 18:45:06.491458 4783 webserver.cc:460] Webserver started at http://127.21.143.67:37647/ using document root <none> and password file <none>
I20260619 18:45:06.492012 4783 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:06.492069 4783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:06.495703 4783 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 18:45:06.497612 4798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:06.498767 4783 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:45:06.498890 4783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/wal
uuid: "b3d01ca389bf4a4fa503d849a23b71c6"
format_stamp: "Formatted at 2026-06-19 18:45:04 on dist-test-slave-mld5"
server_key: "51a67d461aecc2e01b9b9770862d2372"
server_key_iv: "3dd4c89f810e944bad8e5da56f0653bc"
server_key_version: "encryptionkey@0"
I20260619 18:45:06.499460 4783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:06.520326 4783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:06.523126 4783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:06.523370 4783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:06.523948 4783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:45:06.524744 4783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:45:06.524789 4783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:06.524876 4783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:45:06.524909 4783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:06.533375 4783 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.67:39183
I20260619 18:45:06.533397 4911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.67:39183 every 8 connection(s)
I20260619 18:45:06.534401 4783 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
I20260619 18:45:06.538335 4914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:06.536606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.67:54079)
0619 18:45:06.537064 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:06.537181 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":119,"threads_started":1}
I20260619 18:45:06.538594 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 4783
W20260619 18:45:06.538800 4912 heartbeater.cc:644] Failed to heartbeat to 127.21.143.126:42911 (0 consecutive failures): Network error: Failed to ping master at 127.21.143.126:42911: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
I20260619 18:45:06.539405 4914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:06.539158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.67:39681)
0619 18:45:06.539265 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:06.539337 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 18:45:06.540022 4914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:06.539831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.67:42121)
0619 18:45:06.539922 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:06.539967 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 18:45:06.540199 4912 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 18:45:07.223202 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:07.222687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.65:54113)
0619 18:45:07.223022 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:07.223088 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":108,"threads_started":1}
I20260619 18:45:07.350715 4917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:07.350271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.66:50481)
0619 18:45:07.350545 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:07.350606 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":66,"threads_started":1}
I20260619 18:45:07.541004 4918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:07.540636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.67:47199)
0619 18:45:07.540860 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:07.540912 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":62,"threads_started":1}
I20260619 18:45:07.777688 4919 raft_consensus.cc:482] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:45:07.777907 4919 raft_consensus.cc:504] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0eabab97f74bceaf95d7d08feea40c" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 44607 } } peers { permanent_uuid: "b3d01ca389bf4a4fa503d849a23b71c6" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 39183 } } peers { permanent_uuid: "9d8f9ac072db4db8aba2735fd7613aa0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 42281 } }
I20260619 18:45:07.779062 4919 leader_election.cc:288] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7e0eabab97f74bceaf95d7d08feea40c (127.21.143.66:44607), b3d01ca389bf4a4fa503d849a23b71c6 (127.21.143.67:39183)
Jun 19 18:45:07 dist-test-slave-mld5 krb5kdc[3338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894706, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for kudu/127.21.143.66@KRBTEST.COM
Jun 19 18:45:07 dist-test-slave-mld5 krb5kdc[3338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894706, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for kudu/127.21.143.67@KRBTEST.COM
I20260619 18:45:07.796785 4922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:07.779641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:39183 (local address 127.21.143.65:38841)
0619 18:45:07.779894 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:07.779927 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:07.780033 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:07.781944 (+ 1911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:07.781950 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:07.782579 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:45:07.783462 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:07.783485 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:07.785407 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:07.785412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:07.787138 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:07.787150 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:07.788702 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:07.788704 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:07.788766 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:07.789160 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:45:07.789172 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:45:07.791377 (+ 2205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:45:07.793566 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:07.793573 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:07.793589 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:07.794109 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:07.794613 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:07.794616 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:07.794618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:07.794869 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:07.795638 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:45:07.795645 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:45:07.795982 (+ 337us) client_negotiation.cc:769] Sending connection context
0619 18:45:07.796195 (+ 213us) client_negotiation.cc:240] Negotiation successful
0619 18:45:07.796561 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":367,"spinlock_wait_cycles":33280,"thread_start_us":66,"threads_started":1}
I20260619 18:45:07.796816 4921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:07.779489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:38945 (local address 127.21.143.66:44607)
0619 18:45:07.779935 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:07.779964 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:07.780014 (+ 50us) server_negotiation.cc:407] Connection header received
0619 18:45:07.781015 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:07.781025 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:07.781483 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:07.781823 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:07.783649 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:07.785221 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:07.787273 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:07.788044 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:07.788221 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:07.791264 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:07.791269 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:07.791283 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:07.791357 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:07.793445 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:07.794334 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:07.794341 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:07.794347 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:07.794500 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:07.794966 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:07.794970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:07.794972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:07.795516 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:07.795653 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:07.796323 (+ 670us) server_negotiation.cc:299] Negotiation successful
0619 18:45:07.796629 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":120,"threads_started":1}
I20260619 18:45:07.796816 4923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:07.779798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:38841 (local address 127.21.143.67:39183)
0619 18:45:07.780227 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:07.780249 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:07.780286 (+ 37us) server_negotiation.cc:407] Connection header received
0619 18:45:07.781022 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:07.781041 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:07.781483 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:07.781823 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:07.783649 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:07.785229 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:07.787273 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:07.788044 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:07.788526 (+ 482us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:07.791560 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:07.791570 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:07.791585 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:07.791622 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:07.793445 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:07.794334 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:07.794340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:07.794347 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:07.794500 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:07.794966 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:07.794970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:07.794972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:07.795515 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:07.795653 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:07.796323 (+ 670us) server_negotiation.cc:299] Negotiation successful
0619 18:45:07.796629 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":79,"threads_started":1}
I20260619 18:45:07.796785 4920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:07.779264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:44607 (local address 127.21.143.65:38945)
0619 18:45:07.779895 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:07.779909 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:07.780033 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:07.781944 (+ 1911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:07.781950 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:07.782579 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:45:07.783462 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:07.783485 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:07.785407 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:07.785411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:07.787138 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:07.787150 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:07.788184 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:07.788187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:07.788378 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:07.789090 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:45:07.789117 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:45:07.791022 (+ 1905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:45:07.793566 (+ 2544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:07.793573 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:07.793589 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:07.794109 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:07.794613 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:07.794616 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:07.794618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:07.794869 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:07.795638 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:45:07.795645 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:45:07.795982 (+ 337us) client_negotiation.cc:769] Sending connection context
0619 18:45:07.796195 (+ 213us) client_negotiation.cc:240] Negotiation successful
0619 18:45:07.796613 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":489,"spinlock_wait_cycles":62976,"thread_start_us":61,"threads_started":1}
W20260619 18:45:07.799546 4535 leader_election.cc:341] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b3d01ca389bf4a4fa503d849a23b71c6 (127.21.143.67:39183): Not found: Tablet not found: 2a7803e9963543818299d00c2680a9da
W20260619 18:45:07.799750 4535 leader_election.cc:341] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7e0eabab97f74bceaf95d7d08feea40c (127.21.143.66:44607): Not found: Tablet not found: 2a7803e9963543818299d00c2680a9da
I20260619 18:45:07.799813 4535 leader_election.cc:302] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d8f9ac072db4db8aba2735fd7613aa0; no voters: 7e0eabab97f74bceaf95d7d08feea40c, b3d01ca389bf4a4fa503d849a23b71c6
I20260619 18:45:07.800125 4919 raft_consensus.cc:2726] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 18:45:08.224309 4920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:08.223985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.65:42349)
0619 18:45:08.224143 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:08.224208 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 18:45:08.351917 4924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:08.351501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.66:49837)
0619 18:45:08.351754 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:08.351827 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":73,"threads_started":1}
I20260619 18:45:08.542307 4925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:08.541881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.67:33259)
0619 18:45:08.542142 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:08.542203 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":82,"threads_started":1}
I20260619 18:45:09.225566 4926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:09.225106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.65:60233)
0619 18:45:09.225386 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:09.225452 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":81,"threads_started":1}
I20260619 18:45:09.353056 4927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:09.352622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.66:40395)
0619 18:45:09.352894 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:09.352953 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":88,"threads_started":1}
I20260619 18:45:09.417805 4928 raft_consensus.cc:482] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:45:09.417937 4928 raft_consensus.cc:504] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0eabab97f74bceaf95d7d08feea40c" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 44607 } } peers { permanent_uuid: "b3d01ca389bf4a4fa503d849a23b71c6" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 39183 } } peers { permanent_uuid: "9d8f9ac072db4db8aba2735fd7613aa0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 42281 } }
I20260619 18:45:09.418376 4928 leader_election.cc:288] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7e0eabab97f74bceaf95d7d08feea40c (127.21.143.66:44607), b3d01ca389bf4a4fa503d849a23b71c6 (127.21.143.67:39183)
W20260619 18:45:09.419245 4535 leader_election.cc:341] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7e0eabab97f74bceaf95d7d08feea40c (127.21.143.66:44607): Not found: Tablet not found: 2a7803e9963543818299d00c2680a9da
W20260619 18:45:09.419464 4535 leader_election.cc:341] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b3d01ca389bf4a4fa503d849a23b71c6 (127.21.143.67:39183): Not found: Tablet not found: 2a7803e9963543818299d00c2680a9da
I20260619 18:45:09.419525 4535 leader_election.cc:302] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d8f9ac072db4db8aba2735fd7613aa0; no voters: 7e0eabab97f74bceaf95d7d08feea40c, b3d01ca389bf4a4fa503d849a23b71c6
I20260619 18:45:09.419699 4928 raft_consensus.cc:2726] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 18:45:09.543552 4929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:09.543064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.67:36059)
0619 18:45:09.543392 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:09.543462 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":85,"threads_started":1}
I20260619 18:45:10.226716 4930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:10.226300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.65:35903)
0619 18:45:10.226565 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:10.226622 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":74,"threads_started":1}
I20260619 18:45:10.354317 4931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:10.353863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.66:51549)
0619 18:45:10.354152 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:10.354216 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":97,"threads_started":1}
I20260619 18:45:10.544790 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:10.544355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.67:36897)
0619 18:45:10.544620 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:10.544686 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":90,"threads_started":1}
I20260619 18:45:11.228194 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:11.227694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.65:48319)
0619 18:45:11.228018 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:11.228092 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":89,"threads_started":1}
I20260619 18:45:11.355528 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:11.355051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.66:58287)
0619 18:45:11.355372 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:11.355431 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":76,"threads_started":1}
I20260619 18:45:11.545876 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:11.545512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.67:34641)
0619 18:45:11.545738 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:11.545789 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":69,"threads_started":1}
I20260619 18:45:12.229370 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:12.228958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.65:48535)
0619 18:45:12.229214 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:12.229275 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":91,"threads_started":1}
I20260619 18:45:12.356694 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:12.356283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.66:52349)
0619 18:45:12.356526 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:12.356594 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":71,"threads_started":1}
I20260619 18:45:12.547107 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:12.546604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.67:46747)
0619 18:45:12.546939 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:12.547003 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":72,"threads_started":1}
I20260619 18:45:12.627077 4939 raft_consensus.cc:482] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:45:12.627207 4939 raft_consensus.cc:504] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0eabab97f74bceaf95d7d08feea40c" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 44607 } } peers { permanent_uuid: "b3d01ca389bf4a4fa503d849a23b71c6" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 39183 } } peers { permanent_uuid: "9d8f9ac072db4db8aba2735fd7613aa0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 42281 } }
I20260619 18:45:12.627620 4939 leader_election.cc:288] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7e0eabab97f74bceaf95d7d08feea40c (127.21.143.66:44607), b3d01ca389bf4a4fa503d849a23b71c6 (127.21.143.67:39183)
W20260619 18:45:12.628381 4535 leader_election.cc:341] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7e0eabab97f74bceaf95d7d08feea40c (127.21.143.66:44607): Not found: Tablet not found: 2a7803e9963543818299d00c2680a9da
W20260619 18:45:12.628494 4535 leader_election.cc:341] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b3d01ca389bf4a4fa503d849a23b71c6 (127.21.143.67:39183): Not found: Tablet not found: 2a7803e9963543818299d00c2680a9da
I20260619 18:45:12.628541 4535 leader_election.cc:302] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d8f9ac072db4db8aba2735fd7613aa0; no voters: 7e0eabab97f74bceaf95d7d08feea40c, b3d01ca389bf4a4fa503d849a23b71c6
I20260619 18:45:12.628671 4939 raft_consensus.cc:2726] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 18:45:13.230605 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:13.230132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.65:60653)
0619 18:45:13.230442 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:13.230506 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":83,"threads_started":1}
I20260619 18:45:13.357904 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:13.357455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.66:44985)
0619 18:45:13.357742 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:13.357799 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":78,"threads_started":1}
I20260619 18:45:13.548432 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:13.547923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.67:44553)
0619 18:45:13.548269 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:13.548335 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":77,"threads_started":1}
I20260619 18:45:14.232007 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:14.231532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.65:59321)
0619 18:45:14.231840 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:14.231906 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":97,"threads_started":1}
I20260619 18:45:14.358999 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:14.358602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.66:47327)
0619 18:45:14.358849 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:14.358905 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":80,"threads_started":1}
I20260619 18:45:14.389066 4945 raft_consensus.cc:482] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:45:14.389209 4945 raft_consensus.cc:504] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0eabab97f74bceaf95d7d08feea40c" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 44607 } } peers { permanent_uuid: "b3d01ca389bf4a4fa503d849a23b71c6" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 39183 } } peers { permanent_uuid: "9d8f9ac072db4db8aba2735fd7613aa0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 42281 } }
I20260619 18:45:14.389690 4945 leader_election.cc:288] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7e0eabab97f74bceaf95d7d08feea40c (127.21.143.66:44607), b3d01ca389bf4a4fa503d849a23b71c6 (127.21.143.67:39183)
W20260619 18:45:14.390379 4535 leader_election.cc:341] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7e0eabab97f74bceaf95d7d08feea40c (127.21.143.66:44607): Not found: Tablet not found: 2a7803e9963543818299d00c2680a9da
W20260619 18:45:14.390512 4535 leader_election.cc:341] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b3d01ca389bf4a4fa503d849a23b71c6 (127.21.143.67:39183): Not found: Tablet not found: 2a7803e9963543818299d00c2680a9da
I20260619 18:45:14.390570 4535 leader_election.cc:302] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d8f9ac072db4db8aba2735fd7613aa0; no voters: 7e0eabab97f74bceaf95d7d08feea40c, b3d01ca389bf4a4fa503d849a23b71c6
I20260619 18:45:14.390714 4945 raft_consensus.cc:2726] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 18:45:14.549535 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:14.549140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.67:50211)
0619 18:45:14.549381 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:14.549441 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":69,"threads_started":1}
I20260619 18:45:15.233371 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:15.232943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.65:42989)
0619 18:45:15.233212 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:15.233272 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":93,"threads_started":1}
I20260619 18:45:15.360141 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:15.359729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.66:57761)
0619 18:45:15.359984 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:15.360044 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":76,"threads_started":1}
I20260619 18:45:15.550752 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:15.550323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.67:38579)
0619 18:45:15.550575 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:15.550640 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":80,"threads_started":1}
I20260619 18:45:16.234781 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:16.234353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.65:42157)
0619 18:45:16.234619 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:16.234681 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":72,"threads_started":1}
I20260619 18:45:16.361248 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:16.360852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.66:48625)
0619 18:45:16.361102 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:16.361156 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":69,"threads_started":1}
I20260619 18:45:16.551882 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:16.551522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.67:37581)
0619 18:45:16.551742 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:16.551795 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":63,"threads_started":1}
I20260619 18:45:17.236073 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:17.235628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.65:34757)
0619 18:45:17.235909 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:17.235975 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":75,"threads_started":1}
I20260619 18:45:17.362425 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:17.362000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.66:57803)
0619 18:45:17.362265 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:17.362326 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":79,"threads_started":1}
I20260619 18:45:17.553082 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:17.552644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.67:37673)
0619 18:45:17.552913 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:17.552984 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":74,"threads_started":1}
I20260619 18:45:18.237327 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:18.236903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.65:49221)
0619 18:45:18.237167 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:18.237225 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":83,"threads_started":1}
I20260619 18:45:18.363683 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:18.363220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.66:57973)
0619 18:45:18.363518 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:18.363584 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":120,"threads_started":1}
I20260619 18:45:18.554316 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:18.553899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.67:36293)
0619 18:45:18.554151 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:18.554219 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":73,"threads_started":1}
I20260619 18:45:19.238456 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:19.238057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.65:33883)
0619 18:45:19.238293 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:19.238360 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":74,"threads_started":1}
I20260619 18:45:19.365082 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:19.364618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.66:44957)
0619 18:45:19.364922 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:19.364983 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":79,"threads_started":1}
I20260619 18:45:19.555624 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:19.555075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.67:40629)
0619 18:45:19.555470 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:19.555528 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":68,"threads_started":1}
I20260619 18:45:20.239809 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:20.239365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.65:33665)
0619 18:45:20.239643 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:20.239707 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":77,"threads_started":1}
I20260619 18:45:20.366432 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:20.366044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.66:47489)
0619 18:45:20.366286 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:20.366337 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":83,"threads_started":1}
I20260619 18:45:20.556828 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:20.556408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.67:57321)
0619 18:45:20.556663 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:20.556731 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":90,"threads_started":1}
I20260619 18:45:21.241211 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:21.240776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.65:46851)
0619 18:45:21.241033 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:21.241092 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":74,"threads_started":1}
I20260619 18:45:21.367929 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:21.367493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.66:34265)
0619 18:45:21.367751 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:21.367813 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":86,"threads_started":1}
I20260619 18:45:21.374148 4967 raft_consensus.cc:482] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:45:21.374297 4967 raft_consensus.cc:504] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0eabab97f74bceaf95d7d08feea40c" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 44607 } } peers { permanent_uuid: "b3d01ca389bf4a4fa503d849a23b71c6" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 39183 } } peers { permanent_uuid: "9d8f9ac072db4db8aba2735fd7613aa0" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 42281 } }
I20260619 18:45:21.374732 4967 leader_election.cc:288] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7e0eabab97f74bceaf95d7d08feea40c (127.21.143.66:44607), b3d01ca389bf4a4fa503d849a23b71c6 (127.21.143.67:39183)
W20260619 18:45:21.375510 4535 leader_election.cc:341] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b3d01ca389bf4a4fa503d849a23b71c6 (127.21.143.67:39183): Not found: Tablet not found: 2a7803e9963543818299d00c2680a9da
W20260619 18:45:21.375617 4535 leader_election.cc:341] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7e0eabab97f74bceaf95d7d08feea40c (127.21.143.66:44607): Not found: Tablet not found: 2a7803e9963543818299d00c2680a9da
I20260619 18:45:21.375653 4535 leader_election.cc:302] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d8f9ac072db4db8aba2735fd7613aa0; no voters: 7e0eabab97f74bceaf95d7d08feea40c, b3d01ca389bf4a4fa503d849a23b71c6
I20260619 18:45:21.375782 4967 raft_consensus.cc:2726] T 2a7803e9963543818299d00c2680a9da P 9d8f9ac072db4db8aba2735fd7613aa0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 18:45:21.547982 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 4482
I20260619 18:45:21.554610 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 4649
I20260619 18:45:21.558177 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:21.557785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42911 (local address 127.21.143.67:59495)
0619 18:45:21.558034 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:21.558088 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":87,"threads_started":1}
I20260619 18:45:21.560009 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 4783
2026-06-19T18:45:21Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19163 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:45:21 dist-test-slave-mld5 krb5kdc[4971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:45:21 dist-test-slave-mld5 krb5kdc[4971](info): set up 2 sockets
Jun 19 18:45:21 dist-test-slave-mld5 krb5kdc[4971](info): commencing operation
krb5kdc: starting...
W20260619 18:45:22.603593 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:45:22 dist-test-slave-mld5 krb5kdc[4971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:45:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:45:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.143.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
I20260619 18:45:22.720324 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:33503
--webserver_interface=127.21.143.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:39689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:33503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:45:22.833199 4984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:22.833464 4984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:22.833504 4984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:22.838142 4984 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 18:45:22.838194 4984 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 18:45:22.838210 4984 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 18:45:22.838224 4984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:45:22.838243 4984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:22.838260 4984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:22.838275 4984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:45:22.838295 4984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:45:22.843488 4984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:39689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:33503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:33503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:45:22.844507 4984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:22.845578 4984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:22.854071 4989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:22.854072 4990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:22.854187 4992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:22.854267 4984 server_base.cc:1034] running on GCE node
I20260619 18:45:22.854970 4984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:22.856189 4984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:22.857349 4984 hybrid_clock.cc:648] HybridClock initialized: now 1781894722857317 us; error 47 us; skew 500 ppm
Jun 19 18:45:22 dist-test-slave-mld5 krb5kdc[4971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894722, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:45:22.860110 4984 init.cc:377] Logged in from keytab as kudu/127.21.143.126@KRBTEST.COM (short username kudu)
I20260619 18:45:22.861035 4984 webserver.cc:460] Webserver started at http://127.21.143.126:43553/ using document root <none> and password file <none>
I20260619 18:45:22.861552 4984 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:22.861608 4984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:22.861783 4984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:45:22.863894 4984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "751f116d3cd94290a8ab0e974563c3da"
format_stamp: "Formatted at 2026-06-19 18:45:22 on dist-test-slave-mld5"
server_key: "414d21ccf3c6cbcb40dea02ef4ea57b8"
server_key_iv: "070a4a86b2a6ff1cc7fc00888df4d602"
server_key_version: "encryptionkey@0"
I20260619 18:45:22.864413 4984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "751f116d3cd94290a8ab0e974563c3da"
format_stamp: "Formatted at 2026-06-19 18:45:22 on dist-test-slave-mld5"
server_key: "414d21ccf3c6cbcb40dea02ef4ea57b8"
server_key_iv: "070a4a86b2a6ff1cc7fc00888df4d602"
server_key_version: "encryptionkey@0"
I20260619 18:45:22.868150 4984 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:45:22.870571 4999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:22.871883 4984 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:45:22.872009 4984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "751f116d3cd94290a8ab0e974563c3da"
format_stamp: "Formatted at 2026-06-19 18:45:22 on dist-test-slave-mld5"
server_key: "414d21ccf3c6cbcb40dea02ef4ea57b8"
server_key_iv: "070a4a86b2a6ff1cc7fc00888df4d602"
server_key_version: "encryptionkey@0"
I20260619 18:45:22.872113 4984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:22.893736 4984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:22.897352 4984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:22.897579 4984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:22.904917 4984 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:33503
I20260619 18:45:22.905225 5051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:33503 every 8 connection(s)
I20260619 18:45:22.905954 4984 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:45:22.906376 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 4984
I20260619 18:45:22.906517 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:45:22.906831 22077 external_mini_cluster.cc:1428] Setting key 6b670be6d9ece1e16af48a04dec07d92
I20260619 18:45:22.909252 5052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:22.916344 5052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da: Bootstrap starting.
Jun 19 18:45:22 dist-test-slave-mld5 krb5kdc[4971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:45:22.919163 5052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:22.920147 5052 log.cc:824] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:22.922150 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:22.908256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37900 (local address 127.21.143.126:33503)
0619 18:45:22.908688 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:22.908709 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:22.908752 (+ 43us) server_negotiation.cc:407] Connection header received
0619 18:45:22.909640 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:22.909663 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:22.910070 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:22.910432 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:22.911148 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:22.912689 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:22.913567 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:22.914257 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:22.914421 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:22.917034 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:22.917040 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:22.917057 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:22.917087 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:22.918961 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:22.919575 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:22.919581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:22.919587 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:22.919728 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:22.919983 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:22.919989 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:22.919994 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:22.920499 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:22.920680 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:22.920936 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 18:45:22.921249 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":113,"threads_started":1}
I20260619 18:45:22.922434 5052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da: No bootstrap required, opened a new log
I20260619 18:45:22.926046 5052 raft_consensus.cc:348] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751f116d3cd94290a8ab0e974563c3da" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33503 } }
I20260619 18:45:22.926268 5052 raft_consensus.cc:374] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:22.926321 5052 raft_consensus.cc:729] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 751f116d3cd94290a8ab0e974563c3da, State: Initialized, Role: FOLLOWER
I20260619 18:45:22.926774 5052 consensus_queue.cc:260] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751f116d3cd94290a8ab0e974563c3da" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33503 } }
I20260619 18:45:22.926895 5052 raft_consensus.cc:388] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:45:22.926940 5052 raft_consensus.cc:482] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:45:22.927008 5052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:45:22.927891 5052 raft_consensus.cc:504] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751f116d3cd94290a8ab0e974563c3da" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33503 } }
I20260619 18:45:22.928174 5052 leader_election.cc:302] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 751f116d3cd94290a8ab0e974563c3da; no voters:
I20260619 18:45:22.928409 5052 leader_election.cc:288] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:45:22.928560 5057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:45:22.928826 5057 raft_consensus.cc:686] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [term 1 LEADER]: Becoming Leader. State: Replica: 751f116d3cd94290a8ab0e974563c3da, State: Running, Role: LEADER
I20260619 18:45:22.929129 5057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751f116d3cd94290a8ab0e974563c3da" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33503 } }
I20260619 18:45:22.929533 5052 sys_catalog.cc:564] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:45:22.930719 5058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "751f116d3cd94290a8ab0e974563c3da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751f116d3cd94290a8ab0e974563c3da" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33503 } } }
I20260619 18:45:22.930840 5058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [sys.catalog]: This master's current role is: LEADER
I20260619 18:45:22.931157 5059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 751f116d3cd94290a8ab0e974563c3da. Latest consensus state: current_term: 1 leader_uuid: "751f116d3cd94290a8ab0e974563c3da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751f116d3cd94290a8ab0e974563c3da" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33503 } } }
I20260619 18:45:22.931211 5066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:45:22.931284 5059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [sys.catalog]: This master's current role is: LEADER
I20260619 18:45:22.934478 5066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:45:22.939713 5066 catalog_manager.cc:1347] Generated new cluster ID: c3af2a4f86b745b1a552a4997479c295
I20260619 18:45:22.939769 5066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:45:22.966180 5066 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:45:22.968214 5066 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:45:22.978138 5066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da: Generated new TSK 0
I20260619 18:45:22.979110 5066 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.143.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab.
Entry for principal kudu/127.21.143.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab.
I20260619 18:45:23.020171 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.65:0
--local_ip_for_outbound_sockets=127.21.143.65
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:33503
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:39689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:45:23.130805 5078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:23.131089 5078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:23.131161 5078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:23.135179 5078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:23.135244 5078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:23.135362 5078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.65
I20260619 18:45:23.140772 5078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:39689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:33503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.65
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:45:23.141822 5078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:23.142794 5078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:23.152412 5084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:23.152396 5086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:23.152429 5083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:23.152734 5078 server_base.cc:1034] running on GCE node
I20260619 18:45:23.153183 5078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:23.153807 5078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:23.154951 5078 hybrid_clock.cc:648] HybridClock initialized: now 1781894723154920 us; error 43 us; skew 500 ppm
Jun 19 18:45:23 dist-test-slave-mld5 krb5kdc[4971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894723, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:45:23.157617 5078 init.cc:377] Logged in from keytab as kudu/127.21.143.65@KRBTEST.COM (short username kudu)
I20260619 18:45:23.158407 5078 webserver.cc:460] Webserver started at http://127.21.143.65:35489/ using document root <none> and password file <none>
I20260619 18:45:23.158910 5078 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:23.158964 5078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:23.159124 5078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:45:23.161051 5078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/data/instance:
uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c"
format_stamp: "Formatted at 2026-06-19 18:45:23 on dist-test-slave-mld5"
server_key: "41f39e328d937113ab4a1c3f2f150320"
server_key_iv: "538308983a194df44caf9f02ea484543"
server_key_version: "encryptionkey@0"
I20260619 18:45:23.161535 5078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance:
uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c"
format_stamp: "Formatted at 2026-06-19 18:45:23 on dist-test-slave-mld5"
server_key: "41f39e328d937113ab4a1c3f2f150320"
server_key_iv: "538308983a194df44caf9f02ea484543"
server_key_version: "encryptionkey@0"
I20260619 18:45:23.164975 5078 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:45:23.167147 5093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:23.168340 5078 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:45:23.168469 5078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/wal
uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c"
format_stamp: "Formatted at 2026-06-19 18:45:23 on dist-test-slave-mld5"
server_key: "41f39e328d937113ab4a1c3f2f150320"
server_key_iv: "538308983a194df44caf9f02ea484543"
server_key_version: "encryptionkey@0"
I20260619 18:45:23.168576 5078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:23.183271 5078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:23.185983 5078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:23.186192 5078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:23.186776 5078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:45:23.187672 5078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:45:23.187740 5078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:23.187813 5078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:45:23.187855 5078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:23.196449 5078 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.65:44765
I20260619 18:45:23.196470 5206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.65:44765 every 8 connection(s)
I20260619 18:45:23.197367 5078 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
Jun 19 18:45:23 dist-test-slave-mld5 krb5kdc[4971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894723, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:45:23.206549 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 5078
I20260619 18:45:23.206718 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance
I20260619 18:45:23.207013 22077 external_mini_cluster.cc:1428] Setting key 6bd9b418a7b95b3981603615053f290a
I20260619 18:45:23.210846 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:23.199145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:41577 (local address 127.21.143.126:33503)
0619 18:45:23.199454 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:23.199459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:23.200178 (+ 719us) server_negotiation.cc:407] Connection header received
0619 18:45:23.201534 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:23.201537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:23.201585 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:23.201649 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:23.202786 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:23.203373 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:23.204455 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:23.204676 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:23.204747 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:23.207093 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:23.207098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:23.207101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:23.207129 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:23.208765 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:23.209288 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:23.209292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:23.209294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:23.209376 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:23.209731 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:23.209734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:23.209736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:23.209923 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:23.210028 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:23.210532 (+ 504us) server_negotiation.cc:299] Negotiation successful
0619 18:45:23.210670 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221}
I20260619 18:45:23.211977 5209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:23.199555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.65:41577)
0619 18:45:23.200015 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:23.200069 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:23.201329 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:23.201826 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:23.201833 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:23.202184 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:45:23.202654 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:23.202666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:23.203480 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:23.203483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:23.204350 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:23.204359 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:23.204796 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:23.204798 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:23.204938 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:23.205463 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:45:23.205480 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:45:23.206933 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:45:23.208886 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:23.208892 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:23.208907 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:23.209182 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:23.209475 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:23.209478 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:23.209480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:23.209628 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:23.210025 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:45:23.210033 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:45:23.210293 (+ 260us) client_negotiation.cc:769] Sending connection context
0619 18:45:23.210479 (+ 186us) client_negotiation.cc:240] Negotiation successful
0619 18:45:23.210811 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":136,"threads_started":1}
I20260619 18:45:23.213368 5207 heartbeater.cc:347] Connected to a master server at 127.21.143.126:33503
I20260619 18:45:23.213683 5207 heartbeater.cc:464] Registering TS with master...
I20260619 18:45:23.214222 5207 heartbeater.cc:511] Master 127.21.143.126:33503 requested a full tablet report, sending...
I20260619 18:45:23.215891 5016 ts_manager.cc:194] Registered new tserver with Master: 1eafb7aef5ec4fc5bf19fa45bff8fa4c (127.21.143.65:44765)
WARNING: no policy specified for kudu/127.21.143.66@KRBTEST.COM; defaulting to no policy
I20260619 18:45:23.217152 5016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.65@KRBTEST.COM'} at 127.21.143.65:41577
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab.
Entry for principal kudu/127.21.143.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab.
I20260619 18:45:23.232419 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.66:0
--local_ip_for_outbound_sockets=127.21.143.66
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:33503
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:39689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:45:23.343847 5212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:23.344105 5212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:23.344147 5212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:23.348860 5212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:23.348924 5212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:23.349001 5212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.66
I20260619 18:45:23.354307 5212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:39689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:33503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.66
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:45:23.355407 5212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:23.356412 5212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:23.366278 5220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:23.366299 5217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:23.366402 5212 server_base.cc:1034] running on GCE node
W20260619 18:45:23.366299 5218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:23.367022 5212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:23.367730 5212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:23.368889 5212 hybrid_clock.cc:648] HybridClock initialized: now 1781894723368871 us; error 30 us; skew 500 ppm
Jun 19 18:45:23 dist-test-slave-mld5 krb5kdc[4971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894723, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:45:23.371661 5212 init.cc:377] Logged in from keytab as kudu/127.21.143.66@KRBTEST.COM (short username kudu)
I20260619 18:45:23.372576 5212 webserver.cc:460] Webserver started at http://127.21.143.66:46593/ using document root <none> and password file <none>
I20260619 18:45:23.373143 5212 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:23.373205 5212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:23.373366 5212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:45:23.375488 5212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/data/instance:
uuid: "30316ad2efc047989c94abe9712b64cc"
format_stamp: "Formatted at 2026-06-19 18:45:23 on dist-test-slave-mld5"
server_key: "d3a525ebc84a92dfbc548757f3ca1bad"
server_key_iv: "929f71eb58a86a041352bb4f4b080c52"
server_key_version: "encryptionkey@0"
I20260619 18:45:23.376091 5212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance:
uuid: "30316ad2efc047989c94abe9712b64cc"
format_stamp: "Formatted at 2026-06-19 18:45:23 on dist-test-slave-mld5"
server_key: "d3a525ebc84a92dfbc548757f3ca1bad"
server_key_iv: "929f71eb58a86a041352bb4f4b080c52"
server_key_version: "encryptionkey@0"
I20260619 18:45:23.379725 5212 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:45:23.382202 5227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:23.383654 5212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:45:23.383767 5212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/wal
uuid: "30316ad2efc047989c94abe9712b64cc"
format_stamp: "Formatted at 2026-06-19 18:45:23 on dist-test-slave-mld5"
server_key: "d3a525ebc84a92dfbc548757f3ca1bad"
server_key_iv: "929f71eb58a86a041352bb4f4b080c52"
server_key_version: "encryptionkey@0"
I20260619 18:45:23.383869 5212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:23.410054 5212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:23.413101 5212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:23.413261 5212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:23.413816 5212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:45:23.414600 5212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:45:23.414646 5212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:23.414693 5212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:45:23.414713 5212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:23.424166 5212 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.66:42547
I20260619 18:45:23.424190 5340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.66:42547 every 8 connection(s)
I20260619 18:45:23.425129 5212 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
I20260619 18:45:23.429272 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 5212
I20260619 18:45:23.429401 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance
I20260619 18:45:23.429750 22077 external_mini_cluster.cc:1428] Setting key f98f0fc1e260b8f5967ead7dd9e03187
Jun 19 18:45:23 dist-test-slave-mld5 krb5kdc[4971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894723, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.143.67@KRBTEST.COM; defaulting to no policy
I20260619 18:45:23.441342 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:23.427211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:50321 (local address 127.21.143.126:33503)
0619 18:45:23.427428 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:23.427435 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:23.428329 (+ 894us) server_negotiation.cc:407] Connection header received
0619 18:45:23.430101 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:23.430105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:23.430187 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:23.430293 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:23.431826 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:23.432362 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:23.434136 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:23.434357 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:23.434438 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:23.437438 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:23.437442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:23.437452 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:23.437478 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:23.438940 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:23.439713 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:23.439735 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:23.439736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:23.439811 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:23.440182 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:23.440186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:23.440188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:23.440383 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:23.440500 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:23.441036 (+ 536us) server_negotiation.cc:299] Negotiation successful
0619 18:45:23.441168 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.67@KRBTEST.COM" created.
I20260619 18:45:23.442401 5343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:23.427657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.66:50321)
0619 18:45:23.428159 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:23.428212 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:23.429855 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:23.430517 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:23.430526 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:23.430997 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:45:23.431657 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:23.431673 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:23.432493 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:23.432497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:23.433987 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:23.434001 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:23.434518 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:23.434525 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:23.434770 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:23.435525 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:45:23.435555 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:45:23.437264 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:45:23.439069 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:23.439076 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:23.439143 (+ 67us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:23.439595 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:23.439908 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:23.439911 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:23.439913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:23.440070 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:23.440514 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:45:23.440523 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:45:23.440791 (+ 268us) client_negotiation.cc:769] Sending connection context
0619 18:45:23.440980 (+ 189us) client_negotiation.cc:240] Negotiation successful
0619 18:45:23.441370 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":125,"threads_started":1}
I20260619 18:45:23.443653 5341 heartbeater.cc:347] Connected to a master server at 127.21.143.126:33503
I20260619 18:45:23.443920 5341 heartbeater.cc:464] Registering TS with master...
I20260619 18:45:23.444442 5341 heartbeater.cc:511] Master 127.21.143.126:33503 requested a full tablet report, sending...
I20260619 18:45:23.445487 5016 ts_manager.cc:194] Registered new tserver with Master: 30316ad2efc047989c94abe9712b64cc (127.21.143.66:42547)
I20260619 18:45:23.446069 5016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.66@KRBTEST.COM'} at 127.21.143.66:50321
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab.
Entry for principal kudu/127.21.143.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab.
I20260619 18:45:23.456403 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.67:0
--local_ip_for_outbound_sockets=127.21.143.67
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:33503
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:39689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:45:23.564164 5346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:23.564419 5346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:23.564459 5346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:23.568253 5346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:23.568311 5346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:23.568363 5346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.67
I20260619 18:45:23.573278 5346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:39689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:33503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.67
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:45:23.574221 5346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:23.575182 5346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:23.584468 5352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:23.584475 5354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:23.584692 5346 server_base.cc:1034] running on GCE node
W20260619 18:45:23.584472 5351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:23.585237 5346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:23.585815 5346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:23.587021 5346 hybrid_clock.cc:648] HybridClock initialized: now 1781894723587004 us; error 28 us; skew 500 ppm
Jun 19 18:45:23 dist-test-slave-mld5 krb5kdc[4971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894723, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:45:23.589931 5346 init.cc:377] Logged in from keytab as kudu/127.21.143.67@KRBTEST.COM (short username kudu)
I20260619 18:45:23.590880 5346 webserver.cc:460] Webserver started at http://127.21.143.67:45583/ using document root <none> and password file <none>
I20260619 18:45:23.591553 5346 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:23.591645 5346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:23.591882 5346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:45:23.594028 5346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/data/instance:
uuid: "e4216284b2c5484a91ffe9e2d3777c10"
format_stamp: "Formatted at 2026-06-19 18:45:23 on dist-test-slave-mld5"
server_key: "f1f751d8f750aca9e010aca2e621915f"
server_key_iv: "3baa8487be75cc0934c3cbf710cd4b4c"
server_key_version: "encryptionkey@0"
I20260619 18:45:23.594636 5346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance:
uuid: "e4216284b2c5484a91ffe9e2d3777c10"
format_stamp: "Formatted at 2026-06-19 18:45:23 on dist-test-slave-mld5"
server_key: "f1f751d8f750aca9e010aca2e621915f"
server_key_iv: "3baa8487be75cc0934c3cbf710cd4b4c"
server_key_version: "encryptionkey@0"
I20260619 18:45:23.598538 5346 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 18:45:23.601140 5361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:23.602375 5346 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:45:23.602514 5346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/wal
uuid: "e4216284b2c5484a91ffe9e2d3777c10"
format_stamp: "Formatted at 2026-06-19 18:45:23 on dist-test-slave-mld5"
server_key: "f1f751d8f750aca9e010aca2e621915f"
server_key_iv: "3baa8487be75cc0934c3cbf710cd4b4c"
server_key_version: "encryptionkey@0"
I20260619 18:45:23.602643 5346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:23.615687 5346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:23.618420 5346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:23.618611 5346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:23.619164 5346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:45:23.619966 5346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:45:23.620036 5346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:23.620105 5346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:45:23.620151 5346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:23.628816 5346 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.67:37519
I20260619 18:45:23.628835 5474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.67:37519 every 8 connection(s)
I20260619 18:45:23.629803 5346 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
I20260619 18:45:23.632339 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 5346
I20260619 18:45:23.632470 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance
I20260619 18:45:23.632827 22077 external_mini_cluster.cc:1428] Setting key dbdd7bf2dd7a8683ca3a8688cc0bbb75
Jun 19 18:45:23 dist-test-slave-mld5 krb5kdc[4971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894723, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:45:23.643440 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:23.631750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:53021 (local address 127.21.143.126:33503)
0619 18:45:23.631885 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:23.631888 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:23.632785 (+ 897us) server_negotiation.cc:407] Connection header received
0619 18:45:23.634537 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:23.634539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:23.634584 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:23.634656 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:23.635940 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:23.636444 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:23.637577 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:23.637810 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:23.637894 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:23.640232 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:23.640234 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:23.640237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:23.640258 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:23.641684 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:23.642125 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:23.642127 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:23.642129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:23.642198 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:23.642482 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:23.642484 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:23.642485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:23.642672 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:23.642745 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:23.643178 (+ 433us) server_negotiation.cc:299] Negotiation successful
0619 18:45:23.643268 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 18:45:23.644284 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:23.632157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.67:53021)
0619 18:45:23.632611 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:23.632661 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:23.634321 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:23.634838 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:23.634848 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:23.635263 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:45:23.635804 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:23.635817 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:23.636547 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:23.636550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:23.637425 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:23.637433 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:23.637935 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:23.637937 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:23.638069 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:23.638575 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:45:23.638590 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:45:23.640089 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:45:23.641785 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:23.641789 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:23.641800 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:23.642045 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:23.642286 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:23.642288 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:23.642289 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:23.642412 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:23.642771 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:45:23.642776 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:45:23.642986 (+ 210us) client_negotiation.cc:769] Sending connection context
0619 18:45:23.643133 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 18:45:23.643424 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":135,"threads_started":1}
I20260619 18:45:23.645393 5475 heartbeater.cc:347] Connected to a master server at 127.21.143.126:33503
I20260619 18:45:23.645685 5475 heartbeater.cc:464] Registering TS with master...
I20260619 18:45:23.646286 5475 heartbeater.cc:511] Master 127.21.143.126:33503 requested a full tablet report, sending...
I20260619 18:45:23.647279 5016 ts_manager.cc:194] Registered new tserver with Master: e4216284b2c5484a91ffe9e2d3777c10 (127.21.143.67:37519)
I20260619 18:45:23.647856 5016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.67@KRBTEST.COM'} at 127.21.143.67:53021
I20260619 18:45:23.658512 22077 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:45:23.667948 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:23.660636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37914 (local address 127.21.143.126:33503)
0619 18:45:23.660791 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:23.660798 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:23.660962 (+ 164us) server_negotiation.cc:407] Connection header received
0619 18:45:23.661123 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:23.661126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:23.661179 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:23.661262 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:23.661958 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:23.662499 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:23.663361 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:23.663585 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:23.663714 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:23.664972 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:23.664976 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:23.664979 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:23.665005 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:23.666228 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:23.666643 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:23.666646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:23.666648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:23.666750 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:23.667019 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:23.667023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:23.667025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:23.667248 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:23.667405 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:23.667657 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 18:45:23.667781 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 18:45:23.670614 5016 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:45:23.673938 5016 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:45:23.688668 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:23.682362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:44765 (local address 127.0.0.1:38572)
0619 18:45:23.683521 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:23.683574 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:23.683719 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:23.684281 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:23.684284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:23.684318 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:23.684501 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:23.684509 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:23.685957 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:23.685960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:23.687315 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:23.687324 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:23.688152 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:23.688156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:23.688256 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:23.688290 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 18:45:23.688394 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 18:45:23.688504 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"mutex_wait_us":138,"spinlock_wait_cycles":481664,"thread_start_us":124,"threads_started":1}
I20260619 18:45:23.688851 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:23.683045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32894 (local address 127.21.143.66:42547)
0619 18:45:23.683544 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:23.683559 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:23.684010 (+ 451us) server_negotiation.cc:407] Connection header received
0619 18:45:23.684087 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:23.684092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:23.684251 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:23.684383 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:23.684684 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:23.685797 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:23.687289 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:23.688174 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:23.688264 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:23.688391 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:23.688592 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 18:45:23.688699 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":130,"threads_started":1}
I20260619 18:45:23.689033 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:23.682366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53054 (local address 127.21.143.67:37519)
0619 18:45:23.682810 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:23.682822 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:23.683658 (+ 836us) server_negotiation.cc:407] Connection header received
0619 18:45:23.683918 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:23.683925 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:23.684124 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:23.684284 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:23.684798 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:23.685509 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:23.687345 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:23.688158 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:23.688264 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:23.688391 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:23.688785 (+ 394us) server_negotiation.cc:299] Negotiation successful
0619 18:45:23.688887 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":140,"threads_started":1}
I20260619 18:45:23.689211 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:23.682888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:42547 (local address 127.0.0.1:32894)
0619 18:45:23.683917 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:23.683929 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:23.684004 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:23.684477 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:23.684480 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:23.684493 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:23.684577 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:23.684585 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:23.685957 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:23.685960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:23.687136 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:23.687146 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:23.688369 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:23.688375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:23.688456 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:23.688477 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 18:45:23.688547 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 18:45:23.688601 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":90,"spinlock_wait_cycles":53632,"thread_start_us":82,"threads_started":1}
I20260619 18:45:23.689327 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:23.682717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38572 (local address 127.21.143.65:44765)
0619 18:45:23.683029 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:23.683043 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:23.683681 (+ 638us) server_negotiation.cc:407] Connection header received
0619 18:45:23.683890 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:23.683896 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:23.684124 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:23.684460 (+ 336us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:23.684604 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:23.685797 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:23.687421 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:23.688031 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:23.688147 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:23.688289 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:23.689031 (+ 742us) server_negotiation.cc:299] Negotiation successful
0619 18:45:23.689153 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":113,"threads_started":1}
I20260619 18:45:23.689392 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:23.682184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:37519 (local address 127.0.0.1:53054)
0619 18:45:23.683521 (+ 1337us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:23.683572 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:23.683737 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:23.684272 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:23.684278 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:23.684310 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:23.684705 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:23.684708 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:23.685631 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:23.685643 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:23.687172 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:23.687183 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:23.688515 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:23.688539 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:23.688631 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:23.688652 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 18:45:23.689157 (+ 505us) client_negotiation.cc:240] Negotiation successful
0619 18:45:23.689253 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"mutex_wait_us":296,"spinlock_wait_cycles":4992,"thread_start_us":69,"threads_started":1}
I20260619 18:45:23.692028 5275 tablet_service.cc:1467] Processing CreateTablet for tablet a6a9e48ae314434b95dbf85d7bfdf472 (DEFAULT_TABLE table=test-table [id=127c55d0ca7745ff942e73afe62b6106]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:45:23.692134 5141 tablet_service.cc:1467] Processing CreateTablet for tablet a6a9e48ae314434b95dbf85d7bfdf472 (DEFAULT_TABLE table=test-table [id=127c55d0ca7745ff942e73afe62b6106]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:45:23.692137 5409 tablet_service.cc:1467] Processing CreateTablet for tablet a6a9e48ae314434b95dbf85d7bfdf472 (DEFAULT_TABLE table=test-table [id=127c55d0ca7745ff942e73afe62b6106]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:45:23.693075 5275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6a9e48ae314434b95dbf85d7bfdf472. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:23.693125 5141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6a9e48ae314434b95dbf85d7bfdf472. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:23.693128 5409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6a9e48ae314434b95dbf85d7bfdf472. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:23.699285 5492 tablet_bootstrap.cc:492] T a6a9e48ae314434b95dbf85d7bfdf472 P 30316ad2efc047989c94abe9712b64cc: Bootstrap starting.
I20260619 18:45:23.700959 5493 tablet_bootstrap.cc:492] T a6a9e48ae314434b95dbf85d7bfdf472 P e4216284b2c5484a91ffe9e2d3777c10: Bootstrap starting.
I20260619 18:45:23.700961 5494 tablet_bootstrap.cc:492] T a6a9e48ae314434b95dbf85d7bfdf472 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c: Bootstrap starting.
I20260619 18:45:23.701968 5492 tablet_bootstrap.cc:654] T a6a9e48ae314434b95dbf85d7bfdf472 P 30316ad2efc047989c94abe9712b64cc: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:23.702735 5492 log.cc:824] T a6a9e48ae314434b95dbf85d7bfdf472 P 30316ad2efc047989c94abe9712b64cc: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:23.703150 5494 tablet_bootstrap.cc:654] T a6a9e48ae314434b95dbf85d7bfdf472 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:23.703148 5493 tablet_bootstrap.cc:654] T a6a9e48ae314434b95dbf85d7bfdf472 P e4216284b2c5484a91ffe9e2d3777c10: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:23.704504 5493 log.cc:824] T a6a9e48ae314434b95dbf85d7bfdf472 P e4216284b2c5484a91ffe9e2d3777c10: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:23.704504 5494 log.cc:824] T a6a9e48ae314434b95dbf85d7bfdf472 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:23.704784 5492 tablet_bootstrap.cc:492] T a6a9e48ae314434b95dbf85d7bfdf472 P 30316ad2efc047989c94abe9712b64cc: No bootstrap required, opened a new log
I20260619 18:45:23.704980 5492 ts_tablet_manager.cc:1397] T a6a9e48ae314434b95dbf85d7bfdf472 P 30316ad2efc047989c94abe9712b64cc: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 18:45:23.706321 5493 tablet_bootstrap.cc:492] T a6a9e48ae314434b95dbf85d7bfdf472 P e4216284b2c5484a91ffe9e2d3777c10: No bootstrap required, opened a new log
I20260619 18:45:23.706444 5494 tablet_bootstrap.cc:492] T a6a9e48ae314434b95dbf85d7bfdf472 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c: No bootstrap required, opened a new log
I20260619 18:45:23.706467 5493 ts_tablet_manager.cc:1397] T a6a9e48ae314434b95dbf85d7bfdf472 P e4216284b2c5484a91ffe9e2d3777c10: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 18:45:23.706602 5494 ts_tablet_manager.cc:1397] T a6a9e48ae314434b95dbf85d7bfdf472 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 18:45:23.708958 5492 raft_consensus.cc:348] T a6a9e48ae314434b95dbf85d7bfdf472 P 30316ad2efc047989c94abe9712b64cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4216284b2c5484a91ffe9e2d3777c10" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37519 } } peers { permanent_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44765 } } peers { permanent_uuid: "30316ad2efc047989c94abe9712b64cc" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42547 } }
I20260619 18:45:23.709246 5492 raft_consensus.cc:374] T a6a9e48ae314434b95dbf85d7bfdf472 P 30316ad2efc047989c94abe9712b64cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:23.709297 5492 raft_consensus.cc:729] T a6a9e48ae314434b95dbf85d7bfdf472 P 30316ad2efc047989c94abe9712b64cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30316ad2efc047989c94abe9712b64cc, State: Initialized, Role: FOLLOWER
I20260619 18:45:23.709784 5492 consensus_queue.cc:260] T a6a9e48ae314434b95dbf85d7bfdf472 P 30316ad2efc047989c94abe9712b64cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4216284b2c5484a91ffe9e2d3777c10" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37519 } } peers { permanent_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44765 } } peers { permanent_uuid: "30316ad2efc047989c94abe9712b64cc" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42547 } }
I20260619 18:45:23.710510 5493 raft_consensus.cc:348] T a6a9e48ae314434b95dbf85d7bfdf472 P e4216284b2c5484a91ffe9e2d3777c10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4216284b2c5484a91ffe9e2d3777c10" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37519 } } peers { permanent_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44765 } } peers { permanent_uuid: "30316ad2efc047989c94abe9712b64cc" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42547 } }
I20260619 18:45:23.710783 5492 ts_tablet_manager.cc:1428] T a6a9e48ae314434b95dbf85d7bfdf472 P 30316ad2efc047989c94abe9712b64cc: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260619 18:45:23.710796 5493 raft_consensus.cc:374] T a6a9e48ae314434b95dbf85d7bfdf472 P e4216284b2c5484a91ffe9e2d3777c10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:23.710856 5493 raft_consensus.cc:729] T a6a9e48ae314434b95dbf85d7bfdf472 P e4216284b2c5484a91ffe9e2d3777c10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4216284b2c5484a91ffe9e2d3777c10, State: Initialized, Role: FOLLOWER
I20260619 18:45:23.711110 5341 heartbeater.cc:503] Master 127.21.143.126:33503 was elected leader, sending a full tablet report...
I20260619 18:45:23.711372 5493 consensus_queue.cc:260] T a6a9e48ae314434b95dbf85d7bfdf472 P e4216284b2c5484a91ffe9e2d3777c10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4216284b2c5484a91ffe9e2d3777c10" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37519 } } peers { permanent_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44765 } } peers { permanent_uuid: "30316ad2efc047989c94abe9712b64cc" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42547 } }
I20260619 18:45:23.711651 5494 raft_consensus.cc:348] T a6a9e48ae314434b95dbf85d7bfdf472 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4216284b2c5484a91ffe9e2d3777c10" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37519 } } peers { permanent_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44765 } } peers { permanent_uuid: "30316ad2efc047989c94abe9712b64cc" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42547 } }
I20260619 18:45:23.711925 5494 raft_consensus.cc:374] T a6a9e48ae314434b95dbf85d7bfdf472 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:23.712021 5494 raft_consensus.cc:729] T a6a9e48ae314434b95dbf85d7bfdf472 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eafb7aef5ec4fc5bf19fa45bff8fa4c, State: Initialized, Role: FOLLOWER
I20260619 18:45:23.712080 5475 heartbeater.cc:503] Master 127.21.143.126:33503 was elected leader, sending a full tablet report...
I20260619 18:45:23.712543 5494 consensus_queue.cc:260] T a6a9e48ae314434b95dbf85d7bfdf472 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4216284b2c5484a91ffe9e2d3777c10" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37519 } } peers { permanent_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44765 } } peers { permanent_uuid: "30316ad2efc047989c94abe9712b64cc" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42547 } }
I20260619 18:45:23.713613 5494 ts_tablet_manager.cc:1428] T a6a9e48ae314434b95dbf85d7bfdf472 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260619 18:45:23.713979 5207 heartbeater.cc:503] Master 127.21.143.126:33503 was elected leader, sending a full tablet report...
I20260619 18:45:23.715647 5493 ts_tablet_manager.cc:1428] T a6a9e48ae314434b95dbf85d7bfdf472 P e4216284b2c5484a91ffe9e2d3777c10: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260619 18:45:23.735983 5498 raft_consensus.cc:482] T a6a9e48ae314434b95dbf85d7bfdf472 P 30316ad2efc047989c94abe9712b64cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:45:23.736209 5498 raft_consensus.cc:504] T a6a9e48ae314434b95dbf85d7bfdf472 P 30316ad2efc047989c94abe9712b64cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4216284b2c5484a91ffe9e2d3777c10" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37519 } } peers { permanent_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44765 } } peers { permanent_uuid: "30316ad2efc047989c94abe9712b64cc" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42547 } }
I20260619 18:45:23.737422 5498 leader_election.cc:288] T a6a9e48ae314434b95dbf85d7bfdf472 P 30316ad2efc047989c94abe9712b64cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4216284b2c5484a91ffe9e2d3777c10 (127.21.143.67:37519), 1eafb7aef5ec4fc5bf19fa45bff8fa4c (127.21.143.65:44765)
I20260619 18:45:23.741619 5343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:23.737671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:44765 (local address 127.21.143.66:36913)
0619 18:45:23.737782 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:23.737799 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:23.737893 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:23.738199 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:23.738202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:23.738222 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:23.738340 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:23.738345 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:23.739320 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:23.739324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:23.740563 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:23.740573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:23.741200 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:23.741203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:23.741316 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:23.741333 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 18:45:23.741416 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 18:45:23.741487 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 18:45:23.741618 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:23.737752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:36913 (local address 127.21.143.65:44765)
0619 18:45:23.737869 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:23.737873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:23.737887 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:45:23.738022 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:23.738025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:23.738079 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:23.738172 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:23.738462 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:23.739187 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:23.740688 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:23.741059 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:23.741199 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:23.741252 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:23.741444 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 18:45:23.741511 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 18:45:23.741987 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:23.737719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:54807 (local address 127.21.143.67:37519)
0619 18:45:23.737846 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:23.737851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:23.738554 (+ 703us) server_negotiation.cc:407] Connection header received
0619 18:45:23.738664 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:23.738666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:23.738707 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:23.738762 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:23.739010 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:23.739656 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:23.740988 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:23.741386 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:23.741520 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:23.741567 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:23.741783 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 18:45:23.741868 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 18:45:23.742375 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:23.737675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:37519 (local address 127.21.143.66:54807)
0619 18:45:23.738457 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:23.738470 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:23.738567 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:23.738793 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:23.738796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:23.738813 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:23.738914 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:23.738919 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:23.739793 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:23.739796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:23.740864 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:23.740871 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:23.741511 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:23.741515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:23.741615 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:23.741634 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 18:45:23.742172 (+ 538us) client_negotiation.cc:240] Negotiation successful
0619 18:45:23.742242 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"mutex_wait_us":325,"thread_start_us":147,"threads_started":1}
I20260619 18:45:23.742318 5161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a9e48ae314434b95dbf85d7bfdf472" candidate_uuid: "30316ad2efc047989c94abe9712b64cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c" is_pre_election: true
I20260619 18:45:23.742668 5161 raft_consensus.cc:2454] T a6a9e48ae314434b95dbf85d7bfdf472 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30316ad2efc047989c94abe9712b64cc in term 0.
I20260619 18:45:23.743028 5429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a9e48ae314434b95dbf85d7bfdf472" candidate_uuid: "30316ad2efc047989c94abe9712b64cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4216284b2c5484a91ffe9e2d3777c10" is_pre_election: true
I20260619 18:45:23.743218 5228 leader_election.cc:302] T a6a9e48ae314434b95dbf85d7bfdf472 P 30316ad2efc047989c94abe9712b64cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eafb7aef5ec4fc5bf19fa45bff8fa4c, 30316ad2efc047989c94abe9712b64cc; no voters:
I20260619 18:45:23.743357 5429 raft_consensus.cc:2454] T a6a9e48ae314434b95dbf85d7bfdf472 P e4216284b2c5484a91ffe9e2d3777c10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30316ad2efc047989c94abe9712b64cc in term 0.
I20260619 18:45:23.743531 5498 raft_consensus.cc:2781] T a6a9e48ae314434b95dbf85d7bfdf472 P 30316ad2efc047989c94abe9712b64cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:45:23.743629 5498 raft_consensus.cc:482] T a6a9e48ae314434b95dbf85d7bfdf472 P 30316ad2efc047989c94abe9712b64cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:45:23.743690 5498 raft_consensus.cc:3037] T a6a9e48ae314434b95dbf85d7bfdf472 P 30316ad2efc047989c94abe9712b64cc [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:45:23.744798 5498 raft_consensus.cc:504] T a6a9e48ae314434b95dbf85d7bfdf472 P 30316ad2efc047989c94abe9712b64cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4216284b2c5484a91ffe9e2d3777c10" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37519 } } peers { permanent_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44765 } } peers { permanent_uuid: "30316ad2efc047989c94abe9712b64cc" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42547 } }
I20260619 18:45:23.745132 5498 leader_election.cc:288] T a6a9e48ae314434b95dbf85d7bfdf472 P 30316ad2efc047989c94abe9712b64cc [CANDIDATE]: Term 1 election: Requested vote from peers e4216284b2c5484a91ffe9e2d3777c10 (127.21.143.67:37519), 1eafb7aef5ec4fc5bf19fa45bff8fa4c (127.21.143.65:44765)
I20260619 18:45:23.745412 5429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a9e48ae314434b95dbf85d7bfdf472" candidate_uuid: "30316ad2efc047989c94abe9712b64cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4216284b2c5484a91ffe9e2d3777c10"
I20260619 18:45:23.745548 5429 raft_consensus.cc:3037] T a6a9e48ae314434b95dbf85d7bfdf472 P e4216284b2c5484a91ffe9e2d3777c10 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:45:23.745549 5161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a9e48ae314434b95dbf85d7bfdf472" candidate_uuid: "30316ad2efc047989c94abe9712b64cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c"
I20260619 18:45:23.745692 5161 raft_consensus.cc:3037] T a6a9e48ae314434b95dbf85d7bfdf472 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:45:23.746578 5429 raft_consensus.cc:2454] T a6a9e48ae314434b95dbf85d7bfdf472 P e4216284b2c5484a91ffe9e2d3777c10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30316ad2efc047989c94abe9712b64cc in term 1.
I20260619 18:45:23.746742 5161 raft_consensus.cc:2454] T a6a9e48ae314434b95dbf85d7bfdf472 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30316ad2efc047989c94abe9712b64cc in term 1.
I20260619 18:45:23.746904 5229 leader_election.cc:302] T a6a9e48ae314434b95dbf85d7bfdf472 P 30316ad2efc047989c94abe9712b64cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30316ad2efc047989c94abe9712b64cc, e4216284b2c5484a91ffe9e2d3777c10; no voters:
I20260619 18:45:23.747107 5498 raft_consensus.cc:2781] T a6a9e48ae314434b95dbf85d7bfdf472 P 30316ad2efc047989c94abe9712b64cc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:45:23.747283 5498 raft_consensus.cc:686] T a6a9e48ae314434b95dbf85d7bfdf472 P 30316ad2efc047989c94abe9712b64cc [term 1 LEADER]: Becoming Leader. State: Replica: 30316ad2efc047989c94abe9712b64cc, State: Running, Role: LEADER
I20260619 18:45:23.747651 5498 consensus_queue.cc:237] T a6a9e48ae314434b95dbf85d7bfdf472 P 30316ad2efc047989c94abe9712b64cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4216284b2c5484a91ffe9e2d3777c10" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37519 } } peers { permanent_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44765 } } peers { permanent_uuid: "30316ad2efc047989c94abe9712b64cc" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42547 } }
I20260619 18:45:23.750813 5016 catalog_manager.cc:5495] T a6a9e48ae314434b95dbf85d7bfdf472 P 30316ad2efc047989c94abe9712b64cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 30316ad2efc047989c94abe9712b64cc (127.21.143.66). New cstate: current_term: 1 leader_uuid: "30316ad2efc047989c94abe9712b64cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4216284b2c5484a91ffe9e2d3777c10" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30316ad2efc047989c94abe9712b64cc" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42547 } health_report { overall_health: HEALTHY } } }
I20260619 18:45:23.766320 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:23.762853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32900 (local address 127.21.143.66:42547)
0619 18:45:23.762983 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:23.762987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:23.763001 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:45:23.763096 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:23.763098 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:23.763157 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:23.763264 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:45:23.763522 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:23.764025 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:23.764873 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:23.765094 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:23.765216 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:23.765508 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:45:23.765907 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:45:23.766036 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:23.766096 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 18:45:23.766159 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 18:45:23.771965 5161 raft_consensus.cc:1261] T a6a9e48ae314434b95dbf85d7bfdf472 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [term 1 FOLLOWER]: Refusing update from remote peer 30316ad2efc047989c94abe9712b64cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:45:23.771968 5429 raft_consensus.cc:1261] T a6a9e48ae314434b95dbf85d7bfdf472 P e4216284b2c5484a91ffe9e2d3777c10 [term 1 FOLLOWER]: Refusing update from remote peer 30316ad2efc047989c94abe9712b64cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:45:23.772634 5498 consensus_queue.cc:1035] T a6a9e48ae314434b95dbf85d7bfdf472 P 30316ad2efc047989c94abe9712b64cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:45:23.772868 5502 consensus_queue.cc:1035] T a6a9e48ae314434b95dbf85d7bfdf472 P 30316ad2efc047989c94abe9712b64cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4216284b2c5484a91ffe9e2d3777c10" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:45:23.779381 5505 mvcc.cc:205] Tried to move back new op lower bound from 7298640788563070976 to 7298640788473143296. Current Snapshot: MvccSnapshot[applied={T|T < 7298640788563070976}]
I20260619 18:45:23.779764 5507 mvcc.cc:205] Tried to move back new op lower bound from 7298640788563070976 to 7298640788473143296. Current Snapshot: MvccSnapshot[applied={T|T < 7298640788563070976}]
I20260619 18:45:23.790371 5016 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:45:23.790531 5016 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:45:23.792991 5016 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da: Sending DeleteTablet for 3 replicas of tablet a6a9e48ae314434b95dbf85d7bfdf472
I20260619 18:45:23.793849 5141 tablet_service.cc:1514] Processing DeleteTablet for tablet a6a9e48ae314434b95dbf85d7bfdf472 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:45:23 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:38572
I20260619 18:45:23.793852 5409 tablet_service.cc:1514] Processing DeleteTablet for tablet a6a9e48ae314434b95dbf85d7bfdf472 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:45:23 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:53054
I20260619 18:45:23.794400 5516 tablet_replica.cc:330] T a6a9e48ae314434b95dbf85d7bfdf472 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c: stopping tablet replica
I20260619 18:45:23.794406 5515 tablet_replica.cc:330] T a6a9e48ae314434b95dbf85d7bfdf472 P e4216284b2c5484a91ffe9e2d3777c10: stopping tablet replica
I20260619 18:45:23.794420 5275 tablet_service.cc:1514] Processing DeleteTablet for tablet a6a9e48ae314434b95dbf85d7bfdf472 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:45:23 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:32894
I20260619 18:45:23.794727 5515 raft_consensus.cc:2229] T a6a9e48ae314434b95dbf85d7bfdf472 P e4216284b2c5484a91ffe9e2d3777c10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:45:23.794878 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 5078
I20260619 18:45:23.794862 5516 raft_consensus.cc:2229] T a6a9e48ae314434b95dbf85d7bfdf472 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:45:23.795599 5515 raft_consensus.cc:2258] T a6a9e48ae314434b95dbf85d7bfdf472 P e4216284b2c5484a91ffe9e2d3777c10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:45:23.797341 5515 ts_tablet_manager.cc:1905] T a6a9e48ae314434b95dbf85d7bfdf472 P e4216284b2c5484a91ffe9e2d3777c10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:45:23.799512 5517 tablet_replica.cc:330] T a6a9e48ae314434b95dbf85d7bfdf472 P 30316ad2efc047989c94abe9712b64cc: stopping tablet replica
I20260619 18:45:23.799501 5515 ts_tablet_manager.cc:1918] T a6a9e48ae314434b95dbf85d7bfdf472 P e4216284b2c5484a91ffe9e2d3777c10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:45:23.799595 5515 log.cc:1196] T a6a9e48ae314434b95dbf85d7bfdf472 P e4216284b2c5484a91ffe9e2d3777c10: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/wal/wals/a6a9e48ae314434b95dbf85d7bfdf472
I20260619 18:45:23.799937 5515 ts_tablet_manager.cc:1939] T a6a9e48ae314434b95dbf85d7bfdf472 P e4216284b2c5484a91ffe9e2d3777c10: Deleting consensus metadata
I20260619 18:45:23.799957 5517 raft_consensus.cc:2229] T a6a9e48ae314434b95dbf85d7bfdf472 P 30316ad2efc047989c94abe9712b64cc [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:45:23.800478 5517 raft_consensus.cc:2258] T a6a9e48ae314434b95dbf85d7bfdf472 P 30316ad2efc047989c94abe9712b64cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:45:23.801345 5001 catalog_manager.cc:4841] TS e4216284b2c5484a91ffe9e2d3777c10 (127.21.143.67:37519): tablet a6a9e48ae314434b95dbf85d7bfdf472 (table test-table [id=127c55d0ca7745ff942e73afe62b6106]) successfully deleted
I20260619 18:45:23.801887 5517 ts_tablet_manager.cc:1905] T a6a9e48ae314434b95dbf85d7bfdf472 P 30316ad2efc047989c94abe9712b64cc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 18:45:23.802865 5000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.143.65:44765 (error 108)
I20260619 18:45:23.803112 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 5212
I20260619 18:45:23.804062 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:23.803777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:44765 (local address 127.0.0.1:38582)
0619 18:45:23.803941 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:23.803993 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.65:44765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260619 18:45:23.804244 5000 catalog_manager.cc:4584] TS 1eafb7aef5ec4fc5bf19fa45bff8fa4c (127.21.143.65:44765): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a6a9e48ae314434b95dbf85d7bfdf472: Network error: Client connection negotiation failed: client connection to 127.21.143.65:44765: connect: Connection refused (error 111)
I20260619 18:45:23.811164 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 5346
I20260619 18:45:23.813333 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:23.813065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:42547 (local address 127.0.0.1:32906)
0619 18:45:23.813202 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:23.813256 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.66:42547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 18:45:23.816605 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 4984
I20260619 18:45:43.823206 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:33503
--webserver_interface=127.21.143.126
--webserver_port=43553
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:39689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:33503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:45:43.934089 5519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:43.934381 5519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:43.934455 5519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:43.938380 5519 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 18:45:43.938454 5519 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 18:45:43.938482 5519 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 18:45:43.938515 5519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:45:43.938550 5519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:43.938611 5519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:43.938645 5519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:45:43.938683 5519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:45:43.943810 5519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:39689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:33503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:33503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=43553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:45:43.944903 5519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:43.945972 5519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:43.959182 5528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:43.959199 5525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:43.959321 5526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:43.959384 5519 server_base.cc:1034] running on GCE node
I20260619 18:45:43.960150 5519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:43.961280 5519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:43.962445 5519 hybrid_clock.cc:648] HybridClock initialized: now 1781894743962429 us; error 27 us; skew 500 ppm
Jun 19 18:45:43 dist-test-slave-mld5 krb5kdc[4971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894743, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:45:43.965309 5519 init.cc:377] Logged in from keytab as kudu/127.21.143.126@KRBTEST.COM (short username kudu)
I20260619 18:45:43.966223 5519 webserver.cc:460] Webserver started at http://127.21.143.126:43553/ using document root <none> and password file <none>
I20260619 18:45:43.966779 5519 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:43.966866 5519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:43.970594 5519 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:45:43.972730 5535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:43.973863 5519 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:45:43.973994 5519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "751f116d3cd94290a8ab0e974563c3da"
format_stamp: "Formatted at 2026-06-19 18:45:22 on dist-test-slave-mld5"
server_key: "414d21ccf3c6cbcb40dea02ef4ea57b8"
server_key_iv: "070a4a86b2a6ff1cc7fc00888df4d602"
server_key_version: "encryptionkey@0"
I20260619 18:45:43.974567 5519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:43.988277 5519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:43.991166 5519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:43.991421 5519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:43.998487 5519 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:33503
I20260619 18:45:43.998783 5587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:33503 every 8 connection(s)
I20260619 18:45:43.999740 5519 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:45:44.004076 5588 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:45:44.005852 5588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da: Bootstrap starting.
I20260619 18:45:44.009593 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 5519
I20260619 18:45:44.010046 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.65:44765
--local_ip_for_outbound_sockets=127.21.143.65
--tserver_master_addrs=127.21.143.126:33503
--webserver_port=35489
--webserver_interface=127.21.143.65
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:39689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:45:44.023336 5588 log.cc:824] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:44.031708 5588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:45:44.032047 5588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da: Bootstrap complete.
I20260619 18:45:44.035889 5588 raft_consensus.cc:348] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751f116d3cd94290a8ab0e974563c3da" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33503 } }
I20260619 18:45:44.036249 5588 raft_consensus.cc:729] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 751f116d3cd94290a8ab0e974563c3da, State: Initialized, Role: FOLLOWER
I20260619 18:45:44.036769 5588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751f116d3cd94290a8ab0e974563c3da" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33503 } }
I20260619 18:45:44.036883 5588 raft_consensus.cc:388] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:45:44.036986 5588 raft_consensus.cc:482] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:45:44.037117 5588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [term 1 FOLLOWER]: Advancing to term 2
I20260619 18:45:44.038888 5588 raft_consensus.cc:504] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751f116d3cd94290a8ab0e974563c3da" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33503 } }
I20260619 18:45:44.039211 5588 leader_election.cc:302] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 751f116d3cd94290a8ab0e974563c3da; no voters:
I20260619 18:45:44.039561 5588 leader_election.cc:288] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 18:45:44.039670 5592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [term 2 FOLLOWER]: Leader election won for term 2
I20260619 18:45:44.039916 5592 raft_consensus.cc:686] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [term 2 LEADER]: Becoming Leader. State: Replica: 751f116d3cd94290a8ab0e974563c3da, State: Running, Role: LEADER
I20260619 18:45:44.040277 5592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751f116d3cd94290a8ab0e974563c3da" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33503 } }
I20260619 18:45:44.040755 5588 sys_catalog.cc:564] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:45:44.041793 5593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "751f116d3cd94290a8ab0e974563c3da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751f116d3cd94290a8ab0e974563c3da" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33503 } } }
I20260619 18:45:44.041834 5594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 751f116d3cd94290a8ab0e974563c3da. Latest consensus state: current_term: 2 leader_uuid: "751f116d3cd94290a8ab0e974563c3da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751f116d3cd94290a8ab0e974563c3da" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33503 } } }
I20260619 18:45:44.041903 5593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [sys.catalog]: This master's current role is: LEADER
I20260619 18:45:44.041927 5594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [sys.catalog]: This master's current role is: LEADER
I20260619 18:45:44.042379 5601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:45:44.046936 5601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:45:44.047639 5601 catalog_manager.cc:1259] Loaded cluster ID: c3af2a4f86b745b1a552a4997479c295
I20260619 18:45:44.047713 5601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:45:44.049418 5601 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:45:44.049973 5601 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da: Loaded TSK: 0
I20260619 18:45:44.065734 5601 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da: Generated new TSK 1
I20260619 18:45:44.065989 5601 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 18:45:44.136242 5590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:44.136533 5590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:44.136634 5590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:44.141108 5590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:44.141193 5590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:44.141285 5590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.65
I20260619 18:45:44.146687 5590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:39689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.65:44765
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.143.65
--webserver_port=35489
--tserver_master_addrs=127.21.143.126:33503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.65
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:45:44.147725 5590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:44.148772 5590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:44.158154 5616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:44.158154 5615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:44.158154 5618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:44.158684 5590 server_base.cc:1034] running on GCE node
I20260619 18:45:44.159086 5590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:44.159771 5590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:44.160955 5590 hybrid_clock.cc:648] HybridClock initialized: now 1781894744160924 us; error 43 us; skew 500 ppm
Jun 19 18:45:44 dist-test-slave-mld5 krb5kdc[4971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894744, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:45:44.163622 5590 init.cc:377] Logged in from keytab as kudu/127.21.143.65@KRBTEST.COM (short username kudu)
I20260619 18:45:44.164611 5590 webserver.cc:460] Webserver started at http://127.21.143.65:35489/ using document root <none> and password file <none>
I20260619 18:45:44.165141 5590 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:44.165218 5590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:44.169181 5590 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:45:44.171232 5625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:44.172498 5590 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:45:44.172637 5590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/wal
uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c"
format_stamp: "Formatted at 2026-06-19 18:45:23 on dist-test-slave-mld5"
server_key: "41f39e328d937113ab4a1c3f2f150320"
server_key_iv: "538308983a194df44caf9f02ea484543"
server_key_version: "encryptionkey@0"
I20260619 18:45:44.173144 5590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:44.197213 5590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:44.200888 5590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:44.201102 5590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:44.201723 5590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:45:44.202786 5632 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 18:45:44.206796 5590 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 18:45:44.206878 5590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260619 18:45:44.206964 5590 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 18:45:44.208638 5590 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 18:45:44.208709 5590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260619 18:45:44.208948 5632 tablet_bootstrap.cc:492] T a6a9e48ae314434b95dbf85d7bfdf472 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c: Bootstrap starting.
I20260619 18:45:44.219956 5590 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.65:44765
I20260619 18:45:44.220052 5739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.65:44765 every 8 connection(s)
I20260619 18:45:44.221007 5590 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
I20260619 18:45:44.227757 5632 log.cc:824] T a6a9e48ae314434b95dbf85d7bfdf472 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:44.229319 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 5590
I20260619 18:45:44.229765 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.66:42547
--local_ip_for_outbound_sockets=127.21.143.66
--tserver_master_addrs=127.21.143.126:33503
--webserver_port=46593
--webserver_interface=127.21.143.66
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:39689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:45:44 dist-test-slave-mld5 krb5kdc[4971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894744, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:45:44.236406 5632 tablet_bootstrap.cc:492] T a6a9e48ae314434b95dbf85d7bfdf472 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:45:44.236953 5632 tablet_bootstrap.cc:492] T a6a9e48ae314434b95dbf85d7bfdf472 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c: Bootstrap complete.
I20260619 18:45:44.238500 5632 ts_tablet_manager.cc:1397] T a6a9e48ae314434b95dbf85d7bfdf472 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c: Time spent bootstrapping tablet: real 0.030s user 0.024s sys 0.004s
I20260619 18:45:44.238942 5742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:44.223336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:45423 (local address 127.21.143.126:33503)
0619 18:45:44.223748 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:44.223769 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:44.224214 (+ 445us) server_negotiation.cc:407] Connection header received
0619 18:45:44.225646 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:44.225659 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:44.225961 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:44.226235 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:44.227688 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.228740 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.230322 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.231121 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.231451 (+ 330us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:44.233785 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:44.233790 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:44.233801 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:44.233828 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:44.235731 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:44.236261 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:44.236265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:44.236269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:44.236382 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:44.236796 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:44.236799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:44.236801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:44.237192 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:44.237326 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:44.237847 (+ 521us) server_negotiation.cc:299] Negotiation successful
0619 18:45:44.238108 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":119,"threads_started":1}
I20260619 18:45:44.238942 5743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:44.223341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.65:45423)
0619 18:45:44.223993 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:44.224061 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:44.225188 (+ 1127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:44.226369 (+ 1181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:44.226378 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:44.226850 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:45:44.227509 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:44.227529 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.228889 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.228892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:44.230190 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:44.230200 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.231350 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.231357 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:44.231543 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:44.232189 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:45:44.232217 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:45:44.233597 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:45:44.235853 (+ 2256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:44.235858 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:44.235871 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:44.236133 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:44.236479 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:44.236482 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:44.236485 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:44.236690 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:44.237300 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:45:44.237305 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:45:44.237529 (+ 224us) client_negotiation.cc:769] Sending connection context
0619 18:45:44.237719 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 18:45:44.238056 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":154,"threads_started":1}
I20260619 18:45:44.242020 5740 heartbeater.cc:347] Connected to a master server at 127.21.143.126:33503
I20260619 18:45:44.242272 5740 heartbeater.cc:464] Registering TS with master...
I20260619 18:45:44.242813 5740 heartbeater.cc:511] Master 127.21.143.126:33503 requested a full tablet report, sending...
I20260619 18:45:44.242415 5632 raft_consensus.cc:348] T a6a9e48ae314434b95dbf85d7bfdf472 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4216284b2c5484a91ffe9e2d3777c10" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37519 } } peers { permanent_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44765 } } peers { permanent_uuid: "30316ad2efc047989c94abe9712b64cc" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42547 } }
I20260619 18:45:44.242909 5632 raft_consensus.cc:729] T a6a9e48ae314434b95dbf85d7bfdf472 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eafb7aef5ec4fc5bf19fa45bff8fa4c, State: Initialized, Role: FOLLOWER
I20260619 18:45:44.243434 5632 consensus_queue.cc:260] T a6a9e48ae314434b95dbf85d7bfdf472 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4216284b2c5484a91ffe9e2d3777c10" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37519 } } peers { permanent_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44765 } } peers { permanent_uuid: "30316ad2efc047989c94abe9712b64cc" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42547 } }
I20260619 18:45:44.244499 5632 ts_tablet_manager.cc:1428] T a6a9e48ae314434b95dbf85d7bfdf472 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 18:45:44.245900 5552 ts_manager.cc:194] Registered new tserver with Master: 1eafb7aef5ec4fc5bf19fa45bff8fa4c (127.21.143.65:44765)
I20260619 18:45:44.249239 5552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.65@KRBTEST.COM'} at 127.21.143.65:45423
I20260619 18:45:44.251951 5740 heartbeater.cc:503] Master 127.21.143.126:33503 was elected leader, sending a full tablet report...
Jun 19 18:45:44 dist-test-slave-mld5 krb5kdc[4971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894743, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for kudu/127.21.143.65@KRBTEST.COM
I20260619 18:45:44.259261 5747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:44.248587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:44765 (local address 127.0.0.1:35980)
0619 18:45:44.249256 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:44.249301 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:44.249471 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:44.250007 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:44.250013 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:44.250425 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:45:44.250547 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:44.250554 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.251899 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.251902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:44.252821 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:44.252834 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.253542 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.253545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:44.253639 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:44.254103 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:45:44.254116 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:45:44.255627 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:45:44.257559 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:44.257568 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:44.257571 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:44.257782 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:44.258090 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:44.258092 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:44.258094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:44.258167 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:44.258646 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:45:44.258650 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:45:44.258738 (+ 88us) client_negotiation.cc:769] Sending connection context
0619 18:45:44.258866 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 18:45:44.259109 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":109,"threads_started":1}
I20260619 18:45:44.259361 5748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:44.248673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35980 (local address 127.21.143.65:44765)
0619 18:45:44.249003 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:44.249013 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:44.249377 (+ 364us) server_negotiation.cc:407] Connection header received
0619 18:45:44.249648 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:44.249654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:44.249842 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:44.249985 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:44.250672 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.251768 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.252960 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.253429 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.253529 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:44.255760 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:44.255769 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:44.255780 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:44.255805 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:44.257431 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:44.257889 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:44.257895 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:44.257897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:44.257984 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:44.258253 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:44.258258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:44.258260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:44.258520 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:44.258655 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:44.258977 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 18:45:44.259146 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":251,"thread_start_us":91,"threads_started":1}
I20260619 18:45:44.260320 5674 tablet_service.cc:1514] Processing DeleteTablet for tablet a6a9e48ae314434b95dbf85d7bfdf472 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:45:23 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:35980
I20260619 18:45:44.261214 5749 tablet_replica.cc:330] stopping tablet replica
I20260619 18:45:44.261483 5749 raft_consensus.cc:2229] T a6a9e48ae314434b95dbf85d7bfdf472 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:45:44.261729 5749 raft_consensus.cc:2258] T a6a9e48ae314434b95dbf85d7bfdf472 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:45:44.262626 5749 ts_tablet_manager.cc:1905] T a6a9e48ae314434b95dbf85d7bfdf472 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:45:44.266098 5749 ts_tablet_manager.cc:1918] T a6a9e48ae314434b95dbf85d7bfdf472 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:45:44.266206 5749 log.cc:1196] T a6a9e48ae314434b95dbf85d7bfdf472 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/wal/wals/a6a9e48ae314434b95dbf85d7bfdf472
I20260619 18:45:44.266479 5749 ts_tablet_manager.cc:1939] T a6a9e48ae314434b95dbf85d7bfdf472 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c: Deleting consensus metadata
I20260619 18:45:44.268100 5536 catalog_manager.cc:4841] TS 1eafb7aef5ec4fc5bf19fa45bff8fa4c (127.21.143.65:44765): tablet a6a9e48ae314434b95dbf85d7bfdf472 (table test-table [id=127c55d0ca7745ff942e73afe62b6106]) successfully deleted
W20260619 18:45:44.344722 5745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:44.344949 5745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:44.345016 5745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:44.348997 5745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:44.349054 5745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:44.349138 5745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.66
I20260619 18:45:44.354236 5745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:39689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.66:42547
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.143.66
--webserver_port=46593
--tserver_master_addrs=127.21.143.126:33503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.66
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:45:44.355335 5745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:44.356387 5745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:44.366693 5755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:44.366724 5745 server_base.cc:1034] running on GCE node
W20260619 18:45:44.366662 5757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:44.366680 5754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:44.367370 5745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:44.368089 5745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:44.369248 5745 hybrid_clock.cc:648] HybridClock initialized: now 1781894744369234 us; error 28 us; skew 500 ppm
Jun 19 18:45:44 dist-test-slave-mld5 krb5kdc[4971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894744, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:45:44.371963 5745 init.cc:377] Logged in from keytab as kudu/127.21.143.66@KRBTEST.COM (short username kudu)
I20260619 18:45:44.372779 5745 webserver.cc:460] Webserver started at http://127.21.143.66:46593/ using document root <none> and password file <none>
I20260619 18:45:44.373303 5745 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:44.373370 5745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:44.377313 5745 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:45:44.379774 5764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:44.381114 5745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:45:44.381240 5745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/wal
uuid: "30316ad2efc047989c94abe9712b64cc"
format_stamp: "Formatted at 2026-06-19 18:45:23 on dist-test-slave-mld5"
server_key: "d3a525ebc84a92dfbc548757f3ca1bad"
server_key_iv: "929f71eb58a86a041352bb4f4b080c52"
server_key_version: "encryptionkey@0"
I20260619 18:45:44.381784 5745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:44.414362 5745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:44.417361 5745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:44.417586 5745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:44.418197 5745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:45:44.419420 5771 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260619 18:45:44.423152 5771 ts_tablet_manager.cc:1852] T a6a9e48ae314434b95dbf85d7bfdf472 P 30316ad2efc047989c94abe9712b64cc: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260619 18:45:44.423623 5771 ts_tablet_manager.cc:1905] T a6a9e48ae314434b95dbf85d7bfdf472 P 30316ad2efc047989c94abe9712b64cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:45:44.427371 5771 ts_tablet_manager.cc:1918] T a6a9e48ae314434b95dbf85d7bfdf472 P 30316ad2efc047989c94abe9712b64cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:45:44.427464 5771 log.cc:1196] T a6a9e48ae314434b95dbf85d7bfdf472 P 30316ad2efc047989c94abe9712b64cc: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/wal/wals/a6a9e48ae314434b95dbf85d7bfdf472
I20260619 18:45:44.427727 5771 ts_tablet_manager.cc:1939] T a6a9e48ae314434b95dbf85d7bfdf472 P 30316ad2efc047989c94abe9712b64cc: Deleting consensus metadata
I20260619 18:45:44.427979 5745 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260619 18:45:44.428057 5745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260619 18:45:44.428139 5745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:45:44.428171 5745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:44.436810 5745 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.66:42547
I20260619 18:45:44.436841 5878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.66:42547 every 8 connection(s)
I20260619 18:45:44.437770 5745 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
Jun 19 18:45:44 dist-test-slave-mld5 krb5kdc[4971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894744, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:45:44.447362 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 5745
I20260619 18:45:44.447914 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.67:37519
--local_ip_for_outbound_sockets=127.21.143.67
--tserver_master_addrs=127.21.143.126:33503
--webserver_port=45583
--webserver_interface=127.21.143.67
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:39689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:45:44.451149 5742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:44.439457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:56615 (local address 127.21.143.126:33503)
0619 18:45:44.439630 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:44.439635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:44.440498 (+ 863us) server_negotiation.cc:407] Connection header received
0619 18:45:44.441382 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:44.441385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:44.441432 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:44.441522 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:44.442757 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.443451 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.444532 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.444757 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.444839 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:44.447229 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:44.447233 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:44.447235 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:44.447266 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:44.449030 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:44.449587 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:44.449591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:44.449593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:44.449686 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:44.450038 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:44.450043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:44.450045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:44.450235 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:44.450342 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:44.450839 (+ 497us) server_negotiation.cc:299] Negotiation successful
0619 18:45:44.450975 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 18:45:44.451951 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:44.439917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.66:56615)
0619 18:45:44.440351 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:44.440395 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:44.441192 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:44.441681 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:44.441687 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:44.442129 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:45:44.442585 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:44.442599 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.443557 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.443561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:44.444405 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:44.444414 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.444880 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.444882 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:44.445013 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:44.445645 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:45:44.445661 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:45:44.447052 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:45:44.449163 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:44.449168 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:44.449179 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:44.449442 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:44.449790 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:44.449793 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:44.449797 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:44.449938 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:44.450350 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:45:44.450358 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:45:44.450594 (+ 236us) client_negotiation.cc:769] Sending connection context
0619 18:45:44.450781 (+ 187us) client_negotiation.cc:240] Negotiation successful
0619 18:45:44.451108 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":111,"threads_started":1}
I20260619 18:45:44.453164 5879 heartbeater.cc:347] Connected to a master server at 127.21.143.126:33503
I20260619 18:45:44.453437 5879 heartbeater.cc:464] Registering TS with master...
I20260619 18:45:44.454053 5879 heartbeater.cc:511] Master 127.21.143.126:33503 requested a full tablet report, sending...
I20260619 18:45:44.455294 5552 ts_manager.cc:194] Registered new tserver with Master: 30316ad2efc047989c94abe9712b64cc (127.21.143.66:42547)
I20260619 18:45:44.456208 5552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.66@KRBTEST.COM'} at 127.21.143.66:56615
W20260619 18:45:44.563498 5882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:44.563802 5882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:44.563884 5882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:44.568630 5882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:44.568691 5882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:44.568740 5882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.67
I20260619 18:45:44.573640 5882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:39689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.67:37519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.143.67
--webserver_port=45583
--tserver_master_addrs=127.21.143.126:33503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.67
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:45:44.574508 5882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:44.575475 5882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:44.584800 5888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:44.584914 5882 server_base.cc:1034] running on GCE node
W20260619 18:45:44.584801 5890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:44.584800 5887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:44.585441 5882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:44.586093 5882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:44.587273 5882 hybrid_clock.cc:648] HybridClock initialized: now 1781894744587257 us; error 29 us; skew 500 ppm
Jun 19 18:45:44 dist-test-slave-mld5 krb5kdc[4971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894744, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:45:44.589926 5882 init.cc:377] Logged in from keytab as kudu/127.21.143.67@KRBTEST.COM (short username kudu)
I20260619 18:45:44.590718 5882 webserver.cc:460] Webserver started at http://127.21.143.67:45583/ using document root <none> and password file <none>
I20260619 18:45:44.591255 5882 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:44.591398 5882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:44.594960 5882 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 18:45:44.597036 5897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:44.598169 5882 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:45:44.598300 5882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/wal
uuid: "e4216284b2c5484a91ffe9e2d3777c10"
format_stamp: "Formatted at 2026-06-19 18:45:23 on dist-test-slave-mld5"
server_key: "f1f751d8f750aca9e010aca2e621915f"
server_key_iv: "3baa8487be75cc0934c3cbf710cd4b4c"
server_key_version: "encryptionkey@0"
I20260619 18:45:44.598836 5882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:44.633741 5882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:44.636989 5882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:44.637192 5882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:44.637809 5882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:45:44.638603 5882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:45:44.638672 5882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:44.638742 5882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:45:44.638785 5882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:44.647583 5882 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.67:37519
I20260619 18:45:44.647624 6010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.67:37519 every 8 connection(s)
I20260619 18:45:44.648519 5882 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
I20260619 18:45:44.654874 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 5882
Jun 19 18:45:44 dist-test-slave-mld5 krb5kdc[4971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894744, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:45:44.662401 5742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:44.650252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:34623 (local address 127.21.143.126:33503)
0619 18:45:44.650399 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:44.650403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:44.651182 (+ 779us) server_negotiation.cc:407] Connection header received
0619 18:45:44.652472 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:44.652475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:44.652518 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:44.652585 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:44.653659 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.654162 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.655401 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.655709 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.655812 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:44.658394 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:44.658398 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:44.658401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:44.658432 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:44.660199 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:44.660749 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:44.660752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:44.660753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:44.660829 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:44.661233 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:44.661235 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:44.661236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:44.661454 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:44.661552 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:44.662107 (+ 555us) server_negotiation.cc:299] Negotiation successful
0619 18:45:44.662230 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 18:45:44.663203 6013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:44.650617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.67:34623)
0619 18:45:44.651034 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:44.651073 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:44.652281 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:44.652739 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:44.652745 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:44.653079 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:45:44.653538 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:44.653551 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.654261 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.654263 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:44.655274 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:44.655287 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.655868 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.655872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:44.656087 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:44.656659 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:45:44.656683 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:45:44.658207 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:45:44.660310 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:44.660314 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:44.660324 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:44.660639 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:44.660948 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:44.660950 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:44.660951 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:44.661121 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:44.661564 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:45:44.661570 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:45:44.661860 (+ 290us) client_negotiation.cc:769] Sending connection context
0619 18:45:44.662066 (+ 206us) client_negotiation.cc:240] Negotiation successful
0619 18:45:44.662369 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":108,"threads_started":1}
I20260619 18:45:44.663887 6014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:44.655502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53536 (local address 127.21.143.126:33503)
0619 18:45:44.655745 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:44.655751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:44.655996 (+ 245us) server_negotiation.cc:407] Connection header received
0619 18:45:44.656109 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:44.656111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:44.656143 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:44.656208 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:44.657097 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.657581 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.658527 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.658917 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.659070 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:44.660595 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:44.660599 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:44.660602 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:44.660627 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:44.662261 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:44.662723 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:44.662728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:44.662730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:44.662807 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:44.663080 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:44.663084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:44.663086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:44.663241 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:44.663468 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:44.663590 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 18:45:44.663696 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":94,"threads_started":1}
I20260619 18:45:44.664278 6011 heartbeater.cc:347] Connected to a master server at 127.21.143.126:33503
I20260619 18:45:44.664521 6011 heartbeater.cc:464] Registering TS with master...
I20260619 18:45:44.664956 6011 heartbeater.cc:511] Master 127.21.143.126:33503 requested a full tablet report, sending...
I20260619 18:45:44.665951 5552 ts_manager.cc:194] Registered new tserver with Master: e4216284b2c5484a91ffe9e2d3777c10 (127.21.143.67:37519)
I20260619 18:45:44.666448 5552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.67@KRBTEST.COM'} at 127.21.143.67:34623
I20260619 18:45:44.677227 22077 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:45:44.681131 6014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:44.677827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53550 (local address 127.21.143.126:33503)
0619 18:45:44.677957 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:44.677968 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:44.678192 (+ 224us) server_negotiation.cc:407] Connection header received
0619 18:45:44.678329 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:44.678331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:44.678427 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:44.678511 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:45:44.678689 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.679255 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.680144 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.680368 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.680454 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:44.680695 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:45:44.680826 (+ 131us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 18:45:44.681033 (+ 207us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53550: token expired
Metrics: {"server-negotiator.queue_time_us":56}
W20260619 18:45:44.681182 6014 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53550: token expired
W20260619 18:45:44.681525 5478 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.21.143.126:33503)
I20260619 18:45:44.688211 6014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:44.681873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53564 (local address 127.21.143.126:33503)
0619 18:45:44.681987 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:44.681990 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:44.682001 (+ 11us) server_negotiation.cc:407] Connection header received
0619 18:45:44.682051 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:44.682053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:44.682084 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:44.682145 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:44.682674 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.683134 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.683917 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.684113 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.684188 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:44.685471 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:44.685474 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:44.685476 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:44.685497 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:44.686836 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:44.687215 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:44.687218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:44.687219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:44.687290 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:44.687577 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:44.687579 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:44.687580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:44.687734 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:44.687831 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:44.688007 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 18:45:44.688095 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 18:45:44.690464 5552 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:45:44.693764 5552 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:45:44.701650 5674 tablet_service.cc:1467] Processing CreateTablet for tablet 05804420847340f9b0889aacdd79b0f4 (DEFAULT_TABLE table=test-table [id=f26206d1d3b04de9929c8ce053e90d4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:45:44.702379 5674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05804420847340f9b0889aacdd79b0f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:44.705353 5632 tablet_bootstrap.cc:492] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c: Bootstrap starting.
I20260619 18:45:44.706264 5632 tablet_bootstrap.cc:654] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:44.706300 5747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:44.700618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:42547 (local address 127.0.0.1:58072)
0619 18:45:44.700748 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:44.700764 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:44.700835 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:44.701789 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:44.701792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:44.701810 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:44.701908 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:44.701916 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.703584 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.703588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:44.704465 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:44.704474 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.705829 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.705833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:44.705934 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:44.705984 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 18:45:44.706039 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 18:45:44.706120 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 18:45:44.706445 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:44.700860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:37519 (local address 127.0.0.1:34466)
0619 18:45:44.701082 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:44.701096 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:44.701177 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:44.702171 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:44.702174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:44.702200 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:44.702308 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:44.702314 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.703750 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.703753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:44.704948 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:44.704960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.706070 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.706074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:44.706160 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:44.706182 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 18:45:44.706242 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 18:45:44.706310 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":109,"threads_started":1}
I20260619 18:45:44.706534 6022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:44.701298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34466 (local address 127.21.143.67:37519)
0619 18:45:44.701614 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:44.701623 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:44.701641 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:45:44.701703 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:44.701707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:44.701906 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:44.702587 (+ 681us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:44.702762 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.703628 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.705103 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.705941 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.706068 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:44.706214 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:44.706278 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 18:45:44.706396 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":125,"threads_started":1}
I20260619 18:45:44.706960 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:44.700833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58072 (local address 127.21.143.66:42547)
0619 18:45:44.701282 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:44.701294 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:44.701319 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:45:44.701411 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:44.701416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:44.701632 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:44.701807 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:44.702306 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.703473 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.704596 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.705691 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.706464 (+ 773us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:44.706638 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:44.706703 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 18:45:44.706786 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":139,"threads_started":1}
I20260619 18:45:44.708019 5632 tablet_bootstrap.cc:492] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c: No bootstrap required, opened a new log
I20260619 18:45:44.708127 5632 ts_tablet_manager.cc:1397] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 18:45:44.708611 5632 raft_consensus.cc:348] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30316ad2efc047989c94abe9712b64cc" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42547 } } peers { permanent_uuid: "e4216284b2c5484a91ffe9e2d3777c10" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37519 } } peers { permanent_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44765 } }
I20260619 18:45:44.708730 5632 raft_consensus.cc:374] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:44.708777 5632 raft_consensus.cc:729] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eafb7aef5ec4fc5bf19fa45bff8fa4c, State: Initialized, Role: FOLLOWER
I20260619 18:45:44.708532 5813 tablet_service.cc:1467] Processing CreateTablet for tablet 05804420847340f9b0889aacdd79b0f4 (DEFAULT_TABLE table=test-table [id=f26206d1d3b04de9929c8ce053e90d4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:45:44.709110 5632 consensus_queue.cc:260] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30316ad2efc047989c94abe9712b64cc" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42547 } } peers { permanent_uuid: "e4216284b2c5484a91ffe9e2d3777c10" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37519 } } peers { permanent_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44765 } }
I20260619 18:45:44.709585 5813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05804420847340f9b0889aacdd79b0f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:44.709987 5632 ts_tablet_manager.cc:1428] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 18:45:44.709964 5945 tablet_service.cc:1467] Processing CreateTablet for tablet 05804420847340f9b0889aacdd79b0f4 (DEFAULT_TABLE table=test-table [id=f26206d1d3b04de9929c8ce053e90d4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:45:44.710971 5945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05804420847340f9b0889aacdd79b0f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:44.714811 5771 tablet_bootstrap.cc:492] T 05804420847340f9b0889aacdd79b0f4 P 30316ad2efc047989c94abe9712b64cc: Bootstrap starting.
I20260619 18:45:44.716699 5771 tablet_bootstrap.cc:654] T 05804420847340f9b0889aacdd79b0f4 P 30316ad2efc047989c94abe9712b64cc: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:44.717506 5771 log.cc:824] T 05804420847340f9b0889aacdd79b0f4 P 30316ad2efc047989c94abe9712b64cc: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:44.718621 6025 tablet_bootstrap.cc:492] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10: Bootstrap starting.
I20260619 18:45:44.719691 5771 tablet_bootstrap.cc:492] T 05804420847340f9b0889aacdd79b0f4 P 30316ad2efc047989c94abe9712b64cc: No bootstrap required, opened a new log
I20260619 18:45:44.719861 5771 ts_tablet_manager.cc:1397] T 05804420847340f9b0889aacdd79b0f4 P 30316ad2efc047989c94abe9712b64cc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 18:45:44.721182 6025 tablet_bootstrap.cc:654] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:44.722016 6025 log.cc:824] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10: Log is configured to *not* fsync() on all Append() calls
W20260619 18:45:44.722285 5741 tablet.cc:2369] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:45:44.723742 5771 raft_consensus.cc:348] T 05804420847340f9b0889aacdd79b0f4 P 30316ad2efc047989c94abe9712b64cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30316ad2efc047989c94abe9712b64cc" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42547 } } peers { permanent_uuid: "e4216284b2c5484a91ffe9e2d3777c10" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37519 } } peers { permanent_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44765 } }
I20260619 18:45:44.724011 5771 raft_consensus.cc:374] T 05804420847340f9b0889aacdd79b0f4 P 30316ad2efc047989c94abe9712b64cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:44.724071 5771 raft_consensus.cc:729] T 05804420847340f9b0889aacdd79b0f4 P 30316ad2efc047989c94abe9712b64cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30316ad2efc047989c94abe9712b64cc, State: Initialized, Role: FOLLOWER
I20260619 18:45:44.724255 6025 tablet_bootstrap.cc:492] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10: No bootstrap required, opened a new log
I20260619 18:45:44.724467 6025 ts_tablet_manager.cc:1397] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 18:45:44.724563 5771 consensus_queue.cc:260] T 05804420847340f9b0889aacdd79b0f4 P 30316ad2efc047989c94abe9712b64cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30316ad2efc047989c94abe9712b64cc" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42547 } } peers { permanent_uuid: "e4216284b2c5484a91ffe9e2d3777c10" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37519 } } peers { permanent_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44765 } }
I20260619 18:45:44.725199 5879 heartbeater.cc:503] Master 127.21.143.126:33503 was elected leader, sending a full tablet report...
I20260619 18:45:44.725589 5771 ts_tablet_manager.cc:1428] T 05804420847340f9b0889aacdd79b0f4 P 30316ad2efc047989c94abe9712b64cc: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 18:45:44.727906 6025 raft_consensus.cc:348] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30316ad2efc047989c94abe9712b64cc" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42547 } } peers { permanent_uuid: "e4216284b2c5484a91ffe9e2d3777c10" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37519 } } peers { permanent_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44765 } }
I20260619 18:45:44.728171 6025 raft_consensus.cc:374] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:44.728246 6025 raft_consensus.cc:729] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4216284b2c5484a91ffe9e2d3777c10, State: Initialized, Role: FOLLOWER
I20260619 18:45:44.728790 6025 consensus_queue.cc:260] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30316ad2efc047989c94abe9712b64cc" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42547 } } peers { permanent_uuid: "e4216284b2c5484a91ffe9e2d3777c10" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37519 } } peers { permanent_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44765 } }
I20260619 18:45:44.728993 6028 raft_consensus.cc:482] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:45:44.729115 6028 raft_consensus.cc:504] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30316ad2efc047989c94abe9712b64cc" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42547 } } peers { permanent_uuid: "e4216284b2c5484a91ffe9e2d3777c10" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37519 } } peers { permanent_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44765 } }
I20260619 18:45:44.729921 6011 heartbeater.cc:503] Master 127.21.143.126:33503 was elected leader, sending a full tablet report...
I20260619 18:45:44.730257 6025 ts_tablet_manager.cc:1428] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 18:45:44.730885 6028 leader_election.cc:288] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30316ad2efc047989c94abe9712b64cc (127.21.143.66:42547), 1eafb7aef5ec4fc5bf19fa45bff8fa4c (127.21.143.65:44765)
I20260619 18:45:44.734687 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:44.730477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:38737 (local address 127.21.143.66:42547)
0619 18:45:44.730614 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:44.730620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:44.730636 (+ 16us) server_negotiation.cc:407] Connection header received
0619 18:45:44.730787 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:44.730790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:44.730837 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:44.730908 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:44.731292 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.732041 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.733578 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.734202 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.734291 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:44.734332 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:44.734544 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 18:45:44.734598 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 18:45:44.734715 6013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:44.730372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:42547 (local address 127.21.143.67:38737)
0619 18:45:44.730577 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:44.730593 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:44.730676 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:44.731042 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:44.731044 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:44.731060 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:44.731163 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:44.731170 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.732243 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.732246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:44.733444 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:44.733453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.734313 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.734316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:44.734422 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:44.734446 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 18:45:44.734526 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 18:45:44.734593 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260619 18:45:44.735496 5833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05804420847340f9b0889aacdd79b0f4" candidate_uuid: "e4216284b2c5484a91ffe9e2d3777c10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30316ad2efc047989c94abe9712b64cc" is_pre_election: true
I20260619 18:45:44.735675 5748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:44.731267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:48525 (local address 127.21.143.65:44765)
0619 18:45:44.731437 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:44.731441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:44.731648 (+ 207us) server_negotiation.cc:407] Connection header received
0619 18:45:44.731807 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:44.731810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:44.731875 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:44.731967 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:44.732221 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.732788 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.734107 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.734768 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.734879 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:44.734974 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:44.735483 (+ 509us) server_negotiation.cc:299] Negotiation successful
0619 18:45:44.735548 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260619 18:45:44.735689 6030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:44.731170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:44765 (local address 127.21.143.67:48525)
0619 18:45:44.731564 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:44.731577 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:44.731711 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:44.731987 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:44.731990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:44.732005 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:44.732117 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:44.732123 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.732891 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.732895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:44.733999 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:44.734008 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.735134 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.735137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:44.735237 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:44.735262 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 18:45:44.735479 (+ 217us) client_negotiation.cc:240] Negotiation successful
0619 18:45:44.735558 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":85,"threads_started":1}
I20260619 18:45:44.735769 5833 raft_consensus.cc:2454] T 05804420847340f9b0889aacdd79b0f4 P 30316ad2efc047989c94abe9712b64cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4216284b2c5484a91ffe9e2d3777c10 in term 0.
I20260619 18:45:44.736208 5899 leader_election.cc:302] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30316ad2efc047989c94abe9712b64cc, e4216284b2c5484a91ffe9e2d3777c10; no voters:
I20260619 18:45:44.736455 6028 raft_consensus.cc:2781] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:45:44.736536 6028 raft_consensus.cc:482] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:45:44.736569 6028 raft_consensus.cc:3037] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:45:44.736459 5692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05804420847340f9b0889aacdd79b0f4" candidate_uuid: "e4216284b2c5484a91ffe9e2d3777c10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c" is_pre_election: true
I20260619 18:45:44.736764 5692 raft_consensus.cc:2454] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4216284b2c5484a91ffe9e2d3777c10 in term 0.
I20260619 18:45:44.737697 6028 raft_consensus.cc:504] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30316ad2efc047989c94abe9712b64cc" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42547 } } peers { permanent_uuid: "e4216284b2c5484a91ffe9e2d3777c10" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37519 } } peers { permanent_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44765 } }
I20260619 18:45:44.738040 6028 leader_election.cc:288] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10 [CANDIDATE]: Term 1 election: Requested vote from peers 30316ad2efc047989c94abe9712b64cc (127.21.143.66:42547), 1eafb7aef5ec4fc5bf19fa45bff8fa4c (127.21.143.65:44765)
I20260619 18:45:44.738363 5833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05804420847340f9b0889aacdd79b0f4" candidate_uuid: "e4216284b2c5484a91ffe9e2d3777c10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30316ad2efc047989c94abe9712b64cc"
I20260619 18:45:44.738493 5833 raft_consensus.cc:3037] T 05804420847340f9b0889aacdd79b0f4 P 30316ad2efc047989c94abe9712b64cc [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:45:44.738488 5692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05804420847340f9b0889aacdd79b0f4" candidate_uuid: "e4216284b2c5484a91ffe9e2d3777c10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c"
I20260619 18:45:44.738620 5692 raft_consensus.cc:3037] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:45:44.739722 5833 raft_consensus.cc:2454] T 05804420847340f9b0889aacdd79b0f4 P 30316ad2efc047989c94abe9712b64cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4216284b2c5484a91ffe9e2d3777c10 in term 1.
I20260619 18:45:44.739912 5692 raft_consensus.cc:2454] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4216284b2c5484a91ffe9e2d3777c10 in term 1.
I20260619 18:45:44.740032 5899 leader_election.cc:302] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30316ad2efc047989c94abe9712b64cc, e4216284b2c5484a91ffe9e2d3777c10; no voters:
I20260619 18:45:44.740216 6028 raft_consensus.cc:2781] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:45:44.740324 6028 raft_consensus.cc:686] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10 [term 1 LEADER]: Becoming Leader. State: Replica: e4216284b2c5484a91ffe9e2d3777c10, State: Running, Role: LEADER
I20260619 18:45:44.740644 6028 consensus_queue.cc:237] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30316ad2efc047989c94abe9712b64cc" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42547 } } peers { permanent_uuid: "e4216284b2c5484a91ffe9e2d3777c10" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37519 } } peers { permanent_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44765 } }
I20260619 18:45:44.743682 5552 catalog_manager.cc:5495] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4216284b2c5484a91ffe9e2d3777c10 (127.21.143.67). New cstate: current_term: 1 leader_uuid: "e4216284b2c5484a91ffe9e2d3777c10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30316ad2efc047989c94abe9712b64cc" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4216284b2c5484a91ffe9e2d3777c10" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44765 } health_report { overall_health: UNKNOWN } } }
I20260619 18:45:44.757625 6022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:44.754019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34472 (local address 127.21.143.67:37519)
0619 18:45:44.754137 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:44.754142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:44.754195 (+ 53us) server_negotiation.cc:407] Connection header received
0619 18:45:44.754399 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:44.754402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:44.754453 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:44.754562 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:45:44.754865 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.755437 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.756255 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.756500 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.756606 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:44.756895 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:45:44.757264 (+ 369us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:45:44.757402 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:44.757442 (+ 40us) server_negotiation.cc:299] Negotiation successful
0619 18:45:44.757490 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 18:45:44.763199 5692 raft_consensus.cc:1261] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [term 1 FOLLOWER]: Refusing update from remote peer e4216284b2c5484a91ffe9e2d3777c10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:45:44.763199 5833 raft_consensus.cc:1261] T 05804420847340f9b0889aacdd79b0f4 P 30316ad2efc047989c94abe9712b64cc [term 1 FOLLOWER]: Refusing update from remote peer e4216284b2c5484a91ffe9e2d3777c10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:45:44.763863 6028 consensus_queue.cc:1035] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:45:44.764011 6029 consensus_queue.cc:1035] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30316ad2efc047989c94abe9712b64cc" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:45:44.768795 6033 mvcc.cc:205] Tried to move back new op lower bound from 7298640874544033792 to 7298640874460737536. Current Snapshot: MvccSnapshot[applied={T|T < 7298640874544033792}]
I20260619 18:45:44.769822 6034 mvcc.cc:205] Tried to move back new op lower bound from 7298640874544033792 to 7298640874460737536. Current Snapshot: MvccSnapshot[applied={T|T < 7298640874544033792}]
I20260619 18:45:44.771461 6035 mvcc.cc:205] Tried to move back new op lower bound from 7298640874544033792 to 7298640874460737536. Current Snapshot: MvccSnapshot[applied={T|T < 7298640874544033792}]
I20260619 18:45:44.778134 5551 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:45:44.778270 5551 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:45:44.780488 5551 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da: Sending DeleteTablet for 3 replicas of tablet 05804420847340f9b0889aacdd79b0f4
I20260619 18:45:44.781070 5674 tablet_service.cc:1514] Processing DeleteTablet for tablet 05804420847340f9b0889aacdd79b0f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:45:44 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:35980
I20260619 18:45:44.781414 6042 tablet_replica.cc:330] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c: stopping tablet replica
I20260619 18:45:44.781599 6042 raft_consensus.cc:2229] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:45:44.781739 6042 raft_consensus.cc:2258] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:45:44.781823 5813 tablet_service.cc:1514] Processing DeleteTablet for tablet 05804420847340f9b0889aacdd79b0f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:45:44 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:58072
I20260619 18:45:44.782335 6043 tablet_replica.cc:330] T 05804420847340f9b0889aacdd79b0f4 P 30316ad2efc047989c94abe9712b64cc: stopping tablet replica
I20260619 18:45:44.782325 5945 tablet_service.cc:1514] Processing DeleteTablet for tablet 05804420847340f9b0889aacdd79b0f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:45:44 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:34466
I20260619 18:45:44.782577 6042 ts_tablet_manager.cc:1905] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:45:44.782922 6043 raft_consensus.cc:2229] T 05804420847340f9b0889aacdd79b0f4 P 30316ad2efc047989c94abe9712b64cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:45:44.783005 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 5590
I20260619 18:45:44.788086 6043 raft_consensus.cc:2258] T 05804420847340f9b0889aacdd79b0f4 P 30316ad2efc047989c94abe9712b64cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:45:44.788847 6044 tablet_replica.cc:330] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10: stopping tablet replica
W20260619 18:45:44.788964 5536 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.143.65:44765 (error 108)
I20260619 18:45:44.789137 6044 raft_consensus.cc:2229] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:45:44.789575 6044 raft_consensus.cc:2258] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:45:44.789798 6043 ts_tablet_manager.cc:1905] T 05804420847340f9b0889aacdd79b0f4 P 30316ad2efc047989c94abe9712b64cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:45:44.790176 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 5745
I20260619 18:45:44.791579 6044 ts_tablet_manager.cc:1905] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:45:44.791736 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:44.791421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:44765 (local address 127.0.0.1:35996)
0619 18:45:44.791583 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:44.791654 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.65:44765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260619 18:45:44.791905 5536 catalog_manager.cc:4584] TS 1eafb7aef5ec4fc5bf19fa45bff8fa4c (127.21.143.65:44765): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 05804420847340f9b0889aacdd79b0f4: Network error: Client connection negotiation failed: client connection to 127.21.143.65:44765: connect: Connection refused (error 111)
I20260619 18:45:44.793946 6044 ts_tablet_manager.cc:1918] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:45:44.794019 6044 log.cc:1196] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/wal/wals/05804420847340f9b0889aacdd79b0f4
I20260619 18:45:44.794368 6044 ts_tablet_manager.cc:1939] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10: Deleting consensus metadata
I20260619 18:45:44.796010 5537 catalog_manager.cc:4841] TS e4216284b2c5484a91ffe9e2d3777c10 (127.21.143.67:37519): tablet 05804420847340f9b0889aacdd79b0f4 (table test-table [id=f26206d1d3b04de9929c8ce053e90d4f]) successfully deleted
I20260619 18:45:44.796046 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 5882
I20260619 18:45:44.797432 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:44.796283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:42547 (local address 127.0.0.1:58076)
0619 18:45:44.797315 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:44.797365 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.66:42547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":959}
I20260619 18:45:44.801549 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 5519
I20260619 18:46:04.808203 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:33503
--webserver_interface=127.21.143.126
--webserver_port=43553
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:39689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:33503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:04.915107 6046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:04.915423 6046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:04.915503 6046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:04.919445 6046 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 18:46:04.919519 6046 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 18:46:04.919548 6046 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 18:46:04.919584 6046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:04.919621 6046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:04.919668 6046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:04.919710 6046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:04.919745 6046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:04.924888 6046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:39689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:33503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:33503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=43553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:04.925885 6046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:04.926818 6046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:04.935111 6052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:04.935132 6051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:04.935137 6054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:04.935496 6046 server_base.cc:1034] running on GCE node
I20260619 18:46:04.936061 6046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:04.937104 6046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:04.938288 6046 hybrid_clock.cc:648] HybridClock initialized: now 1781894764938281 us; error 28 us; skew 500 ppm
Jun 19 18:46:04 dist-test-slave-mld5 krb5kdc[4971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894764, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:04.940936 6046 init.cc:377] Logged in from keytab as kudu/127.21.143.126@KRBTEST.COM (short username kudu)
I20260619 18:46:04.941752 6046 webserver.cc:460] Webserver started at http://127.21.143.126:43553/ using document root <none> and password file <none>
I20260619 18:46:04.942232 6046 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:04.942288 6046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:04.945721 6046 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:46:04.947688 6061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:04.948859 6046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:46:04.948976 6046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "751f116d3cd94290a8ab0e974563c3da"
format_stamp: "Formatted at 2026-06-19 18:45:22 on dist-test-slave-mld5"
server_key: "414d21ccf3c6cbcb40dea02ef4ea57b8"
server_key_iv: "070a4a86b2a6ff1cc7fc00888df4d602"
server_key_version: "encryptionkey@0"
I20260619 18:46:04.949502 6046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:04.960988 6046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:04.963574 6046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:04.963786 6046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:04.970297 6046 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:33503
I20260619 18:46:04.970570 6113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:33503 every 8 connection(s)
I20260619 18:46:04.971284 6046 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:04.974220 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 6046
I20260619 18:46:04.974668 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.65:44765
--local_ip_for_outbound_sockets=127.21.143.65
--tserver_master_addrs=127.21.143.126:33503
--webserver_port=35489
--webserver_interface=127.21.143.65
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:39689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:46:04.975566 6114 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:46:04.977169 6114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da: Bootstrap starting.
I20260619 18:46:04.995158 6114 log.cc:824] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:05.005995 6114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:46:05.006385 6114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da: Bootstrap complete.
I20260619 18:46:05.010459 6114 raft_consensus.cc:348] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751f116d3cd94290a8ab0e974563c3da" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33503 } }
I20260619 18:46:05.010810 6114 raft_consensus.cc:729] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 751f116d3cd94290a8ab0e974563c3da, State: Initialized, Role: FOLLOWER
I20260619 18:46:05.011276 6114 consensus_queue.cc:260] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751f116d3cd94290a8ab0e974563c3da" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33503 } }
I20260619 18:46:05.011442 6114 raft_consensus.cc:388] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:05.011521 6114 raft_consensus.cc:482] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:05.011623 6114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [term 2 FOLLOWER]: Advancing to term 3
I20260619 18:46:05.013340 6114 raft_consensus.cc:504] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751f116d3cd94290a8ab0e974563c3da" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33503 } }
I20260619 18:46:05.013638 6114 leader_election.cc:302] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 751f116d3cd94290a8ab0e974563c3da; no voters:
I20260619 18:46:05.013938 6114 leader_election.cc:288] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 18:46:05.014015 6118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [term 3 FOLLOWER]: Leader election won for term 3
I20260619 18:46:05.014187 6118 raft_consensus.cc:686] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [term 3 LEADER]: Becoming Leader. State: Replica: 751f116d3cd94290a8ab0e974563c3da, State: Running, Role: LEADER
I20260619 18:46:05.014504 6118 consensus_queue.cc:237] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751f116d3cd94290a8ab0e974563c3da" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33503 } }
I20260619 18:46:05.015086 6114 sys_catalog.cc:564] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:05.015400 6118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 751f116d3cd94290a8ab0e974563c3da. Latest consensus state: current_term: 3 leader_uuid: "751f116d3cd94290a8ab0e974563c3da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751f116d3cd94290a8ab0e974563c3da" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33503 } } }
I20260619 18:46:05.015515 6118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:05.015904 6119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "751f116d3cd94290a8ab0e974563c3da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751f116d3cd94290a8ab0e974563c3da" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 33503 } } }
I20260619 18:46:05.016076 6119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:05.016387 6122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:05.021893 6122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:05.022457 6122 catalog_manager.cc:1259] Loaded cluster ID: c3af2a4f86b745b1a552a4997479c295
I20260619 18:46:05.022523 6122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:05.024827 6122 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:05.025529 6122 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da: Loaded TSK: 0
I20260619 18:46:05.025606 6122 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 751f116d3cd94290a8ab0e974563c3da: Loaded TSK: 1
F20260619 18:46:05.026279 6122 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7efde70829ad google::LogMessage::Fail() at ??:0
@ 0x7efde708588a google::LogMessage::SendToLog() at ??:0
@ 0x7efde70824de google::LogMessage::Flush() at ??:0
@ 0x7efde7083a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7efde9d4618d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7efde9d3ef0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7efdf19e77bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7efdf19c199c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7efdf19c2178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7efdf1a03df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7efdf1a2282b std::function<>::operator()() at ??:0
@ 0x7efdf19c1b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7efdf19c2f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7efdf19bf579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7efdf1a03640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7efde92d43d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7efde92d4c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7efde92d63f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7efde92c5512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7efdea9096db start_thread at ??:0
@ 0x7efde5b8061f clone at ??:0
W20260619 18:46:05.111433 6116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:05.111654 6116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:05.111704 6116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:05.115989 6116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:05.116029 6116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:05.116086 6116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.65
I20260619 18:46:05.122711 6116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:39689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.65:44765
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.143.65
--webserver_port=35489
--tserver_master_addrs=127.21.143.126:33503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.65
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:05.123723 6116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:05.124707 6116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:05.135454 6164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:05.135484 6161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:05.135456 6162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:05.135950 6116 server_base.cc:1034] running on GCE node
I20260619 18:46:05.136386 6116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:05.137157 6116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:05.138371 6116 hybrid_clock.cc:648] HybridClock initialized: now 1781894765138330 us; error 47 us; skew 500 ppm
Jun 19 18:46:05 dist-test-slave-mld5 krb5kdc[4971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894765, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:05.141821 6116 init.cc:377] Logged in from keytab as kudu/127.21.143.65@KRBTEST.COM (short username kudu)
I20260619 18:46:05.142838 6116 webserver.cc:460] Webserver started at http://127.21.143.65:35489/ using document root <none> and password file <none>
I20260619 18:46:05.143549 6116 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:05.143639 6116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:05.148391 6116 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:46:05.151150 6171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:05.152622 6116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:46:05.152745 6116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/wal
uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c"
format_stamp: "Formatted at 2026-06-19 18:45:23 on dist-test-slave-mld5"
server_key: "41f39e328d937113ab4a1c3f2f150320"
server_key_iv: "538308983a194df44caf9f02ea484543"
server_key_version: "encryptionkey@0"
I20260619 18:46:05.153393 6116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:05.177992 6116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:05.181937 6116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:05.182147 6116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:05.182857 6116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:46:05.184276 6178 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 18:46:05.188858 6116 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 18:46:05.188930 6116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260619 18:46:05.188995 6116 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 18:46:05.191181 6116 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 18:46:05.191322 6116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260619 18:46:05.191589 6178 tablet_bootstrap.cc:492] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c: Bootstrap starting.
I20260619 18:46:05.198055 6178 log.cc:824] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:05.201774 6116 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.65:44765
I20260619 18:46:05.201973 6286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.65:44765 every 8 connection(s)
I20260619 18:46:05.202782 6116 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
I20260619 18:46:05.207013 6178 tablet_bootstrap.cc:492] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:46:05.207468 6178 tablet_bootstrap.cc:492] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c: Bootstrap complete.
I20260619 18:46:05.207792 6178 ts_tablet_manager.cc:1397] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.002s
I20260619 18:46:05.209988 6178 raft_consensus.cc:348] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30316ad2efc047989c94abe9712b64cc" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42547 } } peers { permanent_uuid: "e4216284b2c5484a91ffe9e2d3777c10" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37519 } } peers { permanent_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44765 } }
I20260619 18:46:05.210402 6178 raft_consensus.cc:729] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eafb7aef5ec4fc5bf19fa45bff8fa4c, State: Initialized, Role: FOLLOWER
I20260619 18:46:05.210882 6178 consensus_queue.cc:260] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30316ad2efc047989c94abe9712b64cc" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42547 } } peers { permanent_uuid: "e4216284b2c5484a91ffe9e2d3777c10" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37519 } } peers { permanent_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44765 } }
I20260619 18:46:05.211743 6178 ts_tablet_manager.cc:1428] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 18:46:05.212826 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 6116
I20260619 18:46:05.213241 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.66:42547
--local_ip_for_outbound_sockets=127.21.143.66
--tserver_master_addrs=127.21.143.126:33503
--webserver_port=46593
--webserver_interface=127.21.143.66
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:39689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:05.274613 6289 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:05.204822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.65:40837)
0619 18:46:05.205355 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:05.205407 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:05.206395 (+ 988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:05.273616 (+ 67221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":140,"threads_started":1}
W20260619 18:46:05.275256 6287 heartbeater.cc:644] Failed to heartbeat to 127.21.143.126:33503 (0 consecutive failures): Network error: Failed to ping master at 127.21.143.126:33503: Client connection negotiation failed: client connection to 127.21.143.126:33503: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 18:46:05.276038 6289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:05.275747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.65:48589)
0619 18:46:05.275896 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:05.275973 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 18:46:05.276743 6289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:05.276525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.65:41101)
0619 18:46:05.276635 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:05.276687 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 18:46:05.276921 6287 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 18:46:05.326725 6291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:05.327001 6291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:05.327045 6291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:05.331153 6291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:05.331228 6291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:05.331290 6291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.66
I20260619 18:46:05.336588 6291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:39689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.66:42547
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.143.66
--webserver_port=46593
--tserver_master_addrs=127.21.143.126:33503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.66
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:05.337571 6291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:05.338554 6291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:05.348187 6297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:05.348361 6291 server_base.cc:1034] running on GCE node
W20260619 18:46:05.348224 6299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:05.348193 6296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:05.348977 6291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:05.349687 6291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:05.350884 6291 hybrid_clock.cc:648] HybridClock initialized: now 1781894765350870 us; error 30 us; skew 500 ppm
Jun 19 18:46:05 dist-test-slave-mld5 krb5kdc[4971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894765, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:05.353737 6291 init.cc:377] Logged in from keytab as kudu/127.21.143.66@KRBTEST.COM (short username kudu)
I20260619 18:46:05.354652 6291 webserver.cc:460] Webserver started at http://127.21.143.66:46593/ using document root <none> and password file <none>
I20260619 18:46:05.355254 6291 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:05.355378 6291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:05.359586 6291 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260619 18:46:05.361759 6306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:05.362954 6291 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:46:05.363097 6291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/wal
uuid: "30316ad2efc047989c94abe9712b64cc"
format_stamp: "Formatted at 2026-06-19 18:45:23 on dist-test-slave-mld5"
server_key: "d3a525ebc84a92dfbc548757f3ca1bad"
server_key_iv: "929f71eb58a86a041352bb4f4b080c52"
server_key_version: "encryptionkey@0"
I20260619 18:46:05.363713 6291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:05.374837 6291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:05.378235 6291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:05.378448 6291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:05.379067 6291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:46:05.380193 6313 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 18:46:05.384456 6291 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 18:46:05.384541 6291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260619 18:46:05.384627 6291 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 18:46:05.386317 6291 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 18:46:05.386387 6291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 18:46:05.386637 6313 tablet_bootstrap.cc:492] T 05804420847340f9b0889aacdd79b0f4 P 30316ad2efc047989c94abe9712b64cc: Bootstrap starting.
I20260619 18:46:05.392132 6313 log.cc:824] T 05804420847340f9b0889aacdd79b0f4 P 30316ad2efc047989c94abe9712b64cc: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:05.396893 6291 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.66:42547
I20260619 18:46:05.396950 6421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.66:42547 every 8 connection(s)
I20260619 18:46:05.397722 6291 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
I20260619 18:46:05.399873 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 6291
I20260619 18:46:05.400360 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.67:37519
--local_ip_for_outbound_sockets=127.21.143.67
--tserver_master_addrs=127.21.143.126:33503
--webserver_port=45583
--webserver_interface=127.21.143.67
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:39689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:46:05.400593 6313 tablet_bootstrap.cc:492] T 05804420847340f9b0889aacdd79b0f4 P 30316ad2efc047989c94abe9712b64cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:46:05.401016 6313 tablet_bootstrap.cc:492] T 05804420847340f9b0889aacdd79b0f4 P 30316ad2efc047989c94abe9712b64cc: Bootstrap complete.
I20260619 18:46:05.401146 6424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:05.399604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.66:40495)
0619 18:46:05.400106 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:05.400243 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":144,"threads_started":1}
I20260619 18:46:05.401362 6313 ts_tablet_manager.cc:1397] T 05804420847340f9b0889aacdd79b0f4 P 30316ad2efc047989c94abe9712b64cc: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.002s
W20260619 18:46:05.401731 6422 heartbeater.cc:644] Failed to heartbeat to 127.21.143.126:33503 (0 consecutive failures): Network error: Failed to ping master at 127.21.143.126:33503: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
I20260619 18:46:05.402616 6424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:05.402139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.66:37785)
0619 18:46:05.402512 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:05.402556 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":72,"threads_started":1}
I20260619 18:46:05.403235 6424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:05.403034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.66:49955)
0619 18:46:05.403134 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:05.403187 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 18:46:05.403420 6422 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 18:46:05.404868 6313 raft_consensus.cc:348] T 05804420847340f9b0889aacdd79b0f4 P 30316ad2efc047989c94abe9712b64cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30316ad2efc047989c94abe9712b64cc" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42547 } } peers { permanent_uuid: "e4216284b2c5484a91ffe9e2d3777c10" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37519 } } peers { permanent_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44765 } }
I20260619 18:46:05.405251 6313 raft_consensus.cc:729] T 05804420847340f9b0889aacdd79b0f4 P 30316ad2efc047989c94abe9712b64cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30316ad2efc047989c94abe9712b64cc, State: Initialized, Role: FOLLOWER
I20260619 18:46:05.405675 6313 consensus_queue.cc:260] T 05804420847340f9b0889aacdd79b0f4 P 30316ad2efc047989c94abe9712b64cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30316ad2efc047989c94abe9712b64cc" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42547 } } peers { permanent_uuid: "e4216284b2c5484a91ffe9e2d3777c10" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37519 } } peers { permanent_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44765 } }
I20260619 18:46:05.406572 6313 ts_tablet_manager.cc:1428] T 05804420847340f9b0889aacdd79b0f4 P 30316ad2efc047989c94abe9712b64cc: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
I20260619 18:46:05.406920 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:05.406472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.66:55787)
0619 18:46:05.406614 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:05.406668 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 18:46:05.407599 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:05.407405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.66:40631)
0619 18:46:05.407507 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:05.407557 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 18:46:05.515064 6425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:05.515386 6425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:05.515498 6425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:05.519616 6425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:05.519687 6425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:05.519770 6425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.67
I20260619 18:46:05.524941 6425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:39689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.67:37519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.143.67
--webserver_port=45583
--tserver_master_addrs=127.21.143.126:33503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.67
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:05.526062 6425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:05.527102 6425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:05.537005 6432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:05.537005 6433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:05.537045 6435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:05.537214 6425 server_base.cc:1034] running on GCE node
I20260619 18:46:05.537592 6425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:05.538296 6425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:05.539477 6425 hybrid_clock.cc:648] HybridClock initialized: now 1781894765539459 us; error 29 us; skew 500 ppm
Jun 19 18:46:05 dist-test-slave-mld5 krb5kdc[4971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894765, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:05.542150 6425 init.cc:377] Logged in from keytab as kudu/127.21.143.67@KRBTEST.COM (short username kudu)
I20260619 18:46:05.543000 6425 webserver.cc:460] Webserver started at http://127.21.143.67:45583/ using document root <none> and password file <none>
I20260619 18:46:05.543613 6425 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:05.543702 6425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:05.547482 6425 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 18:46:05.549463 6442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:05.550596 6425 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:46:05.550729 6425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/wal
uuid: "e4216284b2c5484a91ffe9e2d3777c10"
format_stamp: "Formatted at 2026-06-19 18:45:23 on dist-test-slave-mld5"
server_key: "f1f751d8f750aca9e010aca2e621915f"
server_key_iv: "3baa8487be75cc0934c3cbf710cd4b4c"
server_key_version: "encryptionkey@0"
I20260619 18:46:05.551270 6425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:05.576028 6425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:05.579383 6425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:05.579591 6425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:05.580183 6425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:46:05.580992 6425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:46:05.581064 6425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:05.581146 6425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:46:05.581183 6425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:05.590179 6425 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.67:37519
I20260619 18:46:05.590199 6555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.67:37519 every 8 connection(s)
I20260619 18:46:05.591233 6425 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
I20260619 18:46:05.595057 6558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:05.593388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.67:59643)
0619 18:46:05.594235 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:05.594342 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":515,"threads_started":1}
W20260619 18:46:05.595605 6556 heartbeater.cc:644] Failed to heartbeat to 127.21.143.126:33503 (0 consecutive failures): Network error: Failed to ping master at 127.21.143.126:33503: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
I20260619 18:46:05.596215 6558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:05.595983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.67:57271)
0619 18:46:05.596107 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:05.596163 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 18:46:05.596834 6558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:05.596666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.67:39903)
0619 18:46:05.596752 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:05.596791 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 18:46:05.597055 6556 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 18:46:05.600050 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 6425
I20260619 18:46:06.277886 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:06.277443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.65:56847)
0619 18:46:06.277717 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:06.277787 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":76,"threads_started":1}
I20260619 18:46:06.408638 6561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:06.408243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.66:44935)
0619 18:46:06.408499 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:06.408552 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":97,"threads_started":1}
I20260619 18:46:06.598070 6562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:06.597666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.67:34911)
0619 18:46:06.597920 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:06.597975 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":93,"threads_started":1}
I20260619 18:46:06.712981 6563 raft_consensus.cc:482] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:46:06.713225 6563 raft_consensus.cc:504] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30316ad2efc047989c94abe9712b64cc" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42547 } } peers { permanent_uuid: "e4216284b2c5484a91ffe9e2d3777c10" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37519 } } peers { permanent_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44765 } }
I20260619 18:46:06.714324 6563 leader_election.cc:288] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30316ad2efc047989c94abe9712b64cc (127.21.143.66:42547), e4216284b2c5484a91ffe9e2d3777c10 (127.21.143.67:37519)
Jun 19 18:46:06 dist-test-slave-mld5 krb5kdc[4971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894765, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for kudu/127.21.143.66@KRBTEST.COM
Jun 19 18:46:06 dist-test-slave-mld5 krb5kdc[4971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894765, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for kudu/127.21.143.67@KRBTEST.COM
I20260619 18:46:06.727707 6566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:06.714976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:47243 (local address 127.21.143.67:37519)
0619 18:46:06.715402 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:06.715419 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:06.715448 (+ 29us) server_negotiation.cc:407] Connection header received
0619 18:46:06.716281 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:06.716288 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:06.716581 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:06.716893 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:06.717489 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:06.718467 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:06.719752 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:06.720215 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:06.720390 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:06.723598 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:06.723606 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:06.723621 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:06.723656 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:06.725435 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:06.725878 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:06.725886 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:06.725892 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:06.726064 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:06.726359 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:06.726363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:06.726365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:06.726735 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:06.726870 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:06.727207 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 18:46:06.727488 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":101,"threads_started":1}
I20260619 18:46:06.727707 6565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:06.714773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:37519 (local address 127.21.143.65:47243)
0619 18:46:06.715068 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:06.715089 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:06.715169 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:06.716824 (+ 1655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:06.716827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:06.717052 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:06.717329 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:06.717343 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:06.718624 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:06.718628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:06.719633 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:06.719643 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:06.720344 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:06.720348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:06.720581 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:06.721275 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:06.721292 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:06.723386 (+ 2094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:06.725555 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:06.725560 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:06.725563 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:06.725759 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:06.726166 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:06.726169 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:06.726170 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:06.726265 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:06.726845 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:06.726848 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:06.726938 (+ 90us) client_negotiation.cc:769] Sending connection context
0619 18:46:06.727102 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 18:46:06.727489 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":150,"spinlock_wait_cycles":9984,"thread_start_us":60,"threads_started":1}
I20260619 18:46:06.727737 6564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:06.714738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:55987 (local address 127.21.143.66:42547)
0619 18:46:06.715091 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:06.715110 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:06.715155 (+ 45us) server_negotiation.cc:407] Connection header received
0619 18:46:06.715474 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:06.715479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:06.716004 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:06.716340 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:06.717497 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:06.718466 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:06.719751 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:06.720214 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:06.720389 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:06.723172 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:06.723178 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:06.723191 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:06.723217 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:06.725098 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:06.725609 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:06.725619 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:06.725623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:06.725738 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:06.726167 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:06.726172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:06.726174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:06.726578 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:06.726728 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:06.727216 (+ 488us) server_negotiation.cc:299] Negotiation successful
0619 18:46:06.727518 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":89,"threads_started":1}
I20260619 18:46:06.727753 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:06.714487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:42547 (local address 127.21.143.65:55987)
0619 18:46:06.714642 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:06.714661 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:06.714813 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:06.716389 (+ 1576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:06.716395 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:06.716864 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:06.717328 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:06.717342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:06.718628 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:06.718631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:06.719632 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:06.719642 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:06.720343 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:06.720346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:06.720580 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:06.721337 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:06.721349 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:06.722971 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:06.725222 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:06.725228 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:06.725242 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:06.725493 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:06.725929 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:06.725932 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:06.725934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:06.726070 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:06.726713 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:06.726719 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:06.726928 (+ 209us) client_negotiation.cc:769] Sending connection context
0619 18:46:06.727102 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 18:46:06.727570 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":303}
I20260619 18:46:06.729182 6376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05804420847340f9b0889aacdd79b0f4" candidate_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "30316ad2efc047989c94abe9712b64cc" is_pre_election: true
I20260619 18:46:06.729527 6376 raft_consensus.cc:2454] T 05804420847340f9b0889aacdd79b0f4 P 30316ad2efc047989c94abe9712b64cc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eafb7aef5ec4fc5bf19fa45bff8fa4c in term 1.
W20260619 18:46:06.730588 6173 leader_election.cc:341] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e4216284b2c5484a91ffe9e2d3777c10 (127.21.143.67:37519): Not found: Tablet not found: 05804420847340f9b0889aacdd79b0f4
I20260619 18:46:06.730831 6173 leader_election.cc:302] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eafb7aef5ec4fc5bf19fa45bff8fa4c, 30316ad2efc047989c94abe9712b64cc; no voters: e4216284b2c5484a91ffe9e2d3777c10
I20260619 18:46:06.731081 6563 raft_consensus.cc:2781] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 18:46:06.731160 6563 raft_consensus.cc:482] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:46:06.731204 6563 raft_consensus.cc:3037] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [term 1 FOLLOWER]: Advancing to term 2
I20260619 18:46:06.733295 6563 raft_consensus.cc:504] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30316ad2efc047989c94abe9712b64cc" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42547 } } peers { permanent_uuid: "e4216284b2c5484a91ffe9e2d3777c10" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37519 } } peers { permanent_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44765 } }
I20260619 18:46:06.733625 6563 leader_election.cc:288] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [CANDIDATE]: Term 2 election: Requested vote from peers 30316ad2efc047989c94abe9712b64cc (127.21.143.66:42547), e4216284b2c5484a91ffe9e2d3777c10 (127.21.143.67:37519)
I20260619 18:46:06.734028 6376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05804420847340f9b0889aacdd79b0f4" candidate_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "30316ad2efc047989c94abe9712b64cc"
I20260619 18:46:06.734160 6376 raft_consensus.cc:3037] T 05804420847340f9b0889aacdd79b0f4 P 30316ad2efc047989c94abe9712b64cc [term 1 FOLLOWER]: Advancing to term 2
W20260619 18:46:06.734241 6173 leader_election.cc:341] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer e4216284b2c5484a91ffe9e2d3777c10 (127.21.143.67:37519): Not found: Tablet not found: 05804420847340f9b0889aacdd79b0f4
I20260619 18:46:06.736313 6376 raft_consensus.cc:2454] T 05804420847340f9b0889aacdd79b0f4 P 30316ad2efc047989c94abe9712b64cc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eafb7aef5ec4fc5bf19fa45bff8fa4c in term 2.
I20260619 18:46:06.736563 6173 leader_election.cc:302] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eafb7aef5ec4fc5bf19fa45bff8fa4c, 30316ad2efc047989c94abe9712b64cc; no voters: e4216284b2c5484a91ffe9e2d3777c10
I20260619 18:46:06.736716 6563 raft_consensus.cc:2781] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [term 2 FOLLOWER]: Leader election won for term 2
I20260619 18:46:06.736929 6563 raft_consensus.cc:686] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [term 2 LEADER]: Becoming Leader. State: Replica: 1eafb7aef5ec4fc5bf19fa45bff8fa4c, State: Running, Role: LEADER
I20260619 18:46:06.737224 6563 consensus_queue.cc:237] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30316ad2efc047989c94abe9712b64cc" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42547 } } peers { permanent_uuid: "e4216284b2c5484a91ffe9e2d3777c10" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37519 } } peers { permanent_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44765 } }
I20260619 18:46:06.737571 6565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:06.737357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.65:60881)
0619 18:46:06.737452 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:06.737499 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 18:46:07.177856 6376 raft_consensus.cc:1261] T 05804420847340f9b0889aacdd79b0f4 P 30316ad2efc047989c94abe9712b64cc [term 2 FOLLOWER]: Refusing update from remote peer 1eafb7aef5ec4fc5bf19fa45bff8fa4c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260619 18:46:07.178715 6563 consensus_queue.cc:1035] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "30316ad2efc047989c94abe9712b64cc" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260619 18:46:07.181121 6570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:07.180581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.66:45535)
0619 18:46:07.180935 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:07.181009 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":107,"threads_started":1}
I20260619 18:46:07.183557 6565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:07.183284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.65:44647)
0619 18:46:07.183414 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:07.183469 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 18:46:07.183753 6173 consensus_peers.cc:487] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c -> Peer e4216284b2c5484a91ffe9e2d3777c10 (127.21.143.67:37519): Couldn't send request to peer e4216284b2c5484a91ffe9e2d3777c10. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05804420847340f9b0889aacdd79b0f4. This is attempt 1: this message will repeat every 5th retry.
I20260619 18:46:07.183955 6570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:07.183281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.66:52025)
0619 18:46:07.183830 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:07.183886 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473}
I20260619 18:46:07.582693 6574 ts_tablet_manager.cc:927] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10: Initiating tablet copy from peer 1eafb7aef5ec4fc5bf19fa45bff8fa4c (127.21.143.65:44765)
I20260619 18:46:07.583282 6574 tablet_copy_client.cc:316] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10: tablet copy: Beginning tablet copy session from remote peer at address 127.21.143.65:44765
Jun 19 18:46:07 dist-test-slave-mld5 krb5kdc[4971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894765, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for kudu/127.21.143.65@KRBTEST.COM
I20260619 18:46:07.593580 6576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:07.583844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:44765 (local address 127.21.143.67:50521)
0619 18:46:07.584126 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:07.584146 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:07.584269 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:07.584779 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:07.584784 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:07.585152 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:07.585278 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:07.585284 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:07.586243 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:07.586245 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:07.586876 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:07.586884 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:07.587471 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:07.587474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:07.587556 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:07.588069 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:07.588079 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:07.589477 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:07.591703 (+ 2226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:07.591712 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:07.591717 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:07.591951 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:07.592260 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:07.592262 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:07.592263 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:07.592340 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:07.592943 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:07.592951 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:07.593099 (+ 148us) client_negotiation.cc:769] Sending connection context
0619 18:46:07.593240 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 18:46:07.593376 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":149,"threads_started":1}
I20260619 18:46:07.593690 6577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:07.584028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:50521 (local address 127.21.143.65:44765)
0619 18:46:07.584346 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:07.584353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:07.584373 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:46:07.584434 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:07.584439 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:07.584645 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:07.584762 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:07.585383 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:07.586150 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:07.586974 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:07.587376 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:07.587457 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:07.589628 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:07.589636 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:07.589648 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:07.589672 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:07.591603 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:07.592059 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:07.592066 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:07.592068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:07.592163 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:07.592447 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:07.592450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:07.592452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:07.592782 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:07.592905 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:07.593343 (+ 438us) server_negotiation.cc:299] Negotiation successful
0619 18:46:07.593514 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":62,"threads_started":1}
I20260619 18:46:07.594915 6256 tablet_copy_service.cc:140] P 1eafb7aef5ec4fc5bf19fa45bff8fa4c: Received BeginTabletCopySession request for tablet 05804420847340f9b0889aacdd79b0f4 from peer e4216284b2c5484a91ffe9e2d3777c10 ({username='kudu', principal='kudu/127.21.143.67@KRBTEST.COM'} at 127.21.143.67:50521)
I20260619 18:46:07.595096 6256 tablet_copy_service.cc:161] P 1eafb7aef5ec4fc5bf19fa45bff8fa4c: Beginning new tablet copy session on tablet 05804420847340f9b0889aacdd79b0f4 from peer e4216284b2c5484a91ffe9e2d3777c10 at {username='kudu', principal='kudu/127.21.143.67@KRBTEST.COM'} at 127.21.143.67:50521: session id = e4216284b2c5484a91ffe9e2d3777c10-05804420847340f9b0889aacdd79b0f4
I20260619 18:46:07.596407 6256 tablet_copy_source_session.cc:205] T 05804420847340f9b0889aacdd79b0f4 P 1eafb7aef5ec4fc5bf19fa45bff8fa4c: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 18:46:07.598927 6576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:07.598717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.67:33379)
0619 18:46:07.598829 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:07.598874 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 18:46:07.600446 6574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05804420847340f9b0889aacdd79b0f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:07.606612 6574 tablet_copy_client.cc:791] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10: tablet copy: Starting download of 0 data blocks...
I20260619 18:46:07.606812 6574 tablet_copy_client.cc:655] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10: tablet copy: Starting download of 1 WAL segments...
I20260619 18:46:07.608402 6574 tablet_copy_client.cc:531] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 18:46:07.609887 6574 tablet_bootstrap.cc:492] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10: Bootstrap starting.
I20260619 18:46:07.615052 6574 log.cc:824] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:07.620970 6574 tablet_bootstrap.cc:492] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:46:07.621263 6574 tablet_bootstrap.cc:492] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10: Bootstrap complete.
I20260619 18:46:07.621562 6574 ts_tablet_manager.cc:1397] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260619 18:46:07.623749 6574 raft_consensus.cc:348] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30316ad2efc047989c94abe9712b64cc" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42547 } } peers { permanent_uuid: "e4216284b2c5484a91ffe9e2d3777c10" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37519 } } peers { permanent_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44765 } }
I20260619 18:46:07.623965 6574 raft_consensus.cc:729] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4216284b2c5484a91ffe9e2d3777c10, State: Initialized, Role: FOLLOWER
I20260619 18:46:07.624408 6574 consensus_queue.cc:260] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30316ad2efc047989c94abe9712b64cc" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42547 } } peers { permanent_uuid: "e4216284b2c5484a91ffe9e2d3777c10" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 37519 } } peers { permanent_uuid: "1eafb7aef5ec4fc5bf19fa45bff8fa4c" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 44765 } }
I20260619 18:46:07.625370 6574 ts_tablet_manager.cc:1428] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10: Time spent starting tablet: real 0.004s user 0.005s sys 0.001s
I20260619 18:46:07.625576 6576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:07.625324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.67:58363)
0619 18:46:07.625453 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:07.625505 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 18:46:07.626031 6256 tablet_copy_service.cc:342] P 1eafb7aef5ec4fc5bf19fa45bff8fa4c: Request end of tablet copy session e4216284b2c5484a91ffe9e2d3777c10-05804420847340f9b0889aacdd79b0f4 received from {username='kudu', principal='kudu/127.21.143.67@KRBTEST.COM'} at 127.21.143.67:50521
I20260619 18:46:07.626142 6256 tablet_copy_service.cc:433] P 1eafb7aef5ec4fc5bf19fa45bff8fa4c: ending tablet copy session e4216284b2c5484a91ffe9e2d3777c10-05804420847340f9b0889aacdd79b0f4 on tablet 05804420847340f9b0889aacdd79b0f4 with peer e4216284b2c5484a91ffe9e2d3777c10
I20260619 18:46:07.626328 6576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:07.626082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.67:42537)
0619 18:46:07.626217 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:07.626270 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 18:46:08.156729 6582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:08.156337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.65:49157)
0619 18:46:08.156558 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:08.156621 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":83,"threads_started":1}
I20260619 18:46:08.156878 6510 raft_consensus.cc:1203] T 05804420847340f9b0889aacdd79b0f4 P e4216284b2c5484a91ffe9e2d3777c10 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260619 18:46:08.158090 6584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:08.157701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.67:53935)
0619 18:46:08.157913 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:08.157978 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":66,"threads_started":1}
I20260619 18:46:08.184957 6585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:08.184592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.66:32981)
0619 18:46:08.184806 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:08.184860 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":62,"threads_started":1}
I20260619 18:46:09.157943 6586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:09.157494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.65:36395)
0619 18:46:09.157771 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:09.157841 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":71,"threads_started":1}
I20260619 18:46:09.159229 6587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:09.158827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.67:43515)
0619 18:46:09.159094 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:09.159150 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":75,"threads_started":1}
I20260619 18:46:09.186218 6588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:09.185755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.66:34375)
0619 18:46:09.186058 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:09.186120 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":73,"threads_started":1}
I20260619 18:46:10.159180 6589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:10.158728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.65:43245)
0619 18:46:10.159006 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:10.159062 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":94,"threads_started":1}
I20260619 18:46:10.160302 6590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:10.159919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.67:35363)
0619 18:46:10.160103 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:10.160224 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":55,"threads_started":1}
I20260619 18:46:10.187608 6591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:10.187052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.66:49573)
0619 18:46:10.187401 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:10.187470 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":114,"threads_started":1}
I20260619 18:46:11.160588 6592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:11.160109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.65:41315)
0619 18:46:11.160409 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:11.160464 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":72,"threads_started":1}
I20260619 18:46:11.161432 6592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:11.161125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.65:39725)
0619 18:46:11.161249 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:11.161339 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 18:46:11.161530 6593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:11.161127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.67:42731)
0619 18:46:11.161362 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:11.161422 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":70,"threads_started":1}
I20260619 18:46:11.188958 6594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:11.188489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.66:54097)
0619 18:46:11.188798 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:11.188852 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":84,"threads_started":1}
I20260619 18:46:12.162678 6596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:12.162212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.67:59203)
0619 18:46:12.162482 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:12.162570 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":58,"threads_started":1}
I20260619 18:46:12.162701 6595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:12.162166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.65:45749)
0619 18:46:12.162540 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:12.162612 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":83,"threads_started":1}
I20260619 18:46:12.190356 6597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:12.189777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.66:35451)
0619 18:46:12.190138 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:12.190226 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":83,"threads_started":1}
I20260619 18:46:13.163933 6598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:13.163502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.67:36635)
0619 18:46:13.163784 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:13.163839 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":97,"threads_started":1}
I20260619 18:46:13.163972 6599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:13.163507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.65:33587)
0619 18:46:13.163838 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:13.163891 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":115,"threads_started":1}
I20260619 18:46:13.191617 6600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:13.191126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.66:60667)
0619 18:46:13.191447 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:13.191508 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":77,"threads_started":1}
I20260619 18:46:14.165066 6601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:14.164650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.67:41543)
0619 18:46:14.164904 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:14.164970 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":95,"threads_started":1}
I20260619 18:46:14.165145 6602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:14.164712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.65:49631)
0619 18:46:14.164980 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:14.165033 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":123,"threads_started":1}
I20260619 18:46:14.192737 6603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:14.192331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.66:55105)
0619 18:46:14.192565 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:14.192619 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":79,"threads_started":1}
I20260619 18:46:15.166318 6604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:15.165800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.67:56479)
0619 18:46:15.166098 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:15.166166 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":109,"threads_started":1}
I20260619 18:46:15.166337 6605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:15.165912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.65:34395)
0619 18:46:15.166145 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:15.166196 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":83,"threads_started":1}
I20260619 18:46:15.193814 6606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:15.193422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.66:53861)
0619 18:46:15.193670 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:15.193721 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":79,"threads_started":1}
I20260619 18:46:16.167793 6607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:16.167358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.65:46355)
0619 18:46:16.167637 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:16.167696 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":73,"threads_started":1}
I20260619 18:46:16.167876 6608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:16.167365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.67:43487)
0619 18:46:16.167680 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:16.167776 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":93,"threads_started":1}
I20260619 18:46:16.195181 6609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:16.194727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.66:47209)
0619 18:46:16.195007 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:16.195071 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":76,"threads_started":1}
I20260619 18:46:17.168982 6611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:17.168505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.65:38031)
0619 18:46:17.168817 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:17.168889 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":77,"threads_started":1}
I20260619 18:46:17.169227 6612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:17.168658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.67:41833)
0619 18:46:17.168928 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:17.168998 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":65,"threads_started":1}
I20260619 18:46:17.196563 6613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:17.196173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.66:54119)
0619 18:46:17.196412 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:17.196468 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":75,"threads_started":1}
I20260619 18:46:18.170089 6614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:18.169660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.65:56909)
0619 18:46:18.169937 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:18.169990 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":102,"threads_started":1}
I20260619 18:46:18.170333 6615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:18.169954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.67:37591)
0619 18:46:18.170177 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:18.170224 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":81,"threads_started":1}
I20260619 18:46:18.197621 6616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:18.197227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.66:56799)
0619 18:46:18.197477 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:18.197528 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":85,"threads_started":1}
I20260619 18:46:19.171591 6617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:19.171105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.67:45467)
0619 18:46:19.171456 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:19.171506 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":86,"threads_started":1}
I20260619 18:46:19.171590 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:19.171105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.65:47263)
0619 18:46:19.171432 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:19.171490 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":83,"threads_started":1}
I20260619 18:46:19.198812 6619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:19.198401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.66:52483)
0619 18:46:19.198652 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:19.198714 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":73,"threads_started":1}
I20260619 18:46:20.172844 6620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:20.172444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.67:37117)
0619 18:46:20.172692 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:20.172753 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":66,"threads_started":1}
I20260619 18:46:20.172888 6621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:20.172467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.65:34865)
0619 18:46:20.172742 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:20.172797 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":106,"threads_started":1}
I20260619 18:46:20.200060 6622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:20.199672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:33503 (local address 127.21.143.66:38079)
0619 18:46:20.199911 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:20.199966 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.126:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":69,"threads_started":1}
I20260619 18:46:20.601922 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 6116
I20260619 18:46:20.607713 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 6291
I20260619 18:46:20.612972 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 6425
2026-06-19T18:46:20Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59053 ms)
[----------] 30 tests from SecurityITest (238034 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:20 dist-test-slave-mld5 krb5kdc[6625](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:20 dist-test-slave-mld5 krb5kdc[6625](info): set up 2 sockets
Jun 19 18:46:20 dist-test-slave-mld5 krb5kdc[6625](info): commencing operation
krb5kdc: starting...
W20260619 18:46:21.658986 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:46:21 dist-test-slave-mld5 krb5kdc[6625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.143.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
I20260619 18:46:21.772274 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:38961
--webserver_interface=127.21.143.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:40673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:38961
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:21.880030 6638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:21.880308 6638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:21.880374 6638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:21.884354 6638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:21.884430 6638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:21.884469 6638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:21.884524 6638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:21.884572 6638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:21.889678 6638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:40673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:38961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:38961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:21.890734 6638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:21.891769 6638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:21.900154 6644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:21.900169 6643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:21.900312 6638 server_base.cc:1034] running on GCE node
W20260619 18:46:21.900480 6646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:21.901026 6638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:21.902015 6638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:21.903173 6638 hybrid_clock.cc:648] HybridClock initialized: now 1781894781903149 us; error 36 us; skew 500 ppm
Jun 19 18:46:21 dist-test-slave-mld5 krb5kdc[6625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894781, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:21.905792 6638 init.cc:377] Logged in from keytab as kudu/127.21.143.126@KRBTEST.COM (short username kudu)
I20260619 18:46:21.906821 6638 webserver.cc:460] Webserver started at http://127.21.143.126:39319/ using document root <none> and password file <none>
I20260619 18:46:21.907511 6638 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:21.907579 6638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:21.907747 6638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:21.909720 6638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "5f44e1a96ce34db9a86e45ba67680e19"
format_stamp: "Formatted at 2026-06-19 18:46:21 on dist-test-slave-mld5"
server_key: "fc67fa69ec3a8ef31c825443c892c6e8"
server_key_iv: "781d7ea675473610e8e0048bb9cead4f"
server_key_version: "encryptionkey@0"
I20260619 18:46:21.910233 6638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "5f44e1a96ce34db9a86e45ba67680e19"
format_stamp: "Formatted at 2026-06-19 18:46:21 on dist-test-slave-mld5"
server_key: "fc67fa69ec3a8ef31c825443c892c6e8"
server_key_iv: "781d7ea675473610e8e0048bb9cead4f"
server_key_version: "encryptionkey@0"
I20260619 18:46:21.913834 6638 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:46:21.916180 6653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:21.917348 6638 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:46:21.917441 6638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "5f44e1a96ce34db9a86e45ba67680e19"
format_stamp: "Formatted at 2026-06-19 18:46:21 on dist-test-slave-mld5"
server_key: "fc67fa69ec3a8ef31c825443c892c6e8"
server_key_iv: "781d7ea675473610e8e0048bb9cead4f"
server_key_version: "encryptionkey@0"
I20260619 18:46:21.917609 6638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:21.935446 6638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:21.938798 6638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:21.939005 6638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:21.946148 6638 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:38961
I20260619 18:46:21.946166 6705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:38961 every 8 connection(s)
I20260619 18:46:21.947135 6638 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:21.948052 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 6638
I20260619 18:46:21.948205 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:46:21.948498 22077 external_mini_cluster.cc:1428] Setting key d64dd043c610a4d936a87e69e2b8ecc2
I20260619 18:46:21.950335 6706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:21.955900 6706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f44e1a96ce34db9a86e45ba67680e19: Bootstrap starting.
Jun 19 18:46:21 dist-test-slave-mld5 krb5kdc[6625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:46:21.958426 6706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f44e1a96ce34db9a86e45ba67680e19: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:21.959260 6706 log.cc:824] T 00000000000000000000000000000000 P 5f44e1a96ce34db9a86e45ba67680e19: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:21.961524 6706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f44e1a96ce34db9a86e45ba67680e19: No bootstrap required, opened a new log
I20260619 18:46:21.962414 6709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:21.949871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43792 (local address 127.21.143.126:38961)
0619 18:46:21.950280 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:21.950293 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:21.950326 (+ 33us) server_negotiation.cc:407] Connection header received
0619 18:46:21.950990 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:21.951006 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:21.951371 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:21.951700 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:21.952499 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:21.953448 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:21.954332 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:21.954793 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:21.954962 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:21.957820 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:21.957825 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:21.957836 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:21.957863 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:21.959649 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:21.960116 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:21.960121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:21.960125 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:21.960228 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:21.960512 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:21.960514 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:21.960515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:21.960983 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:21.961127 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:21.961401 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 18:46:21.961693 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":131,"threads_started":1}
I20260619 18:46:21.965358 6706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f44e1a96ce34db9a86e45ba67680e19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f44e1a96ce34db9a86e45ba67680e19" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 38961 } }
I20260619 18:46:21.965600 6706 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f44e1a96ce34db9a86e45ba67680e19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:21.965677 6706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f44e1a96ce34db9a86e45ba67680e19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f44e1a96ce34db9a86e45ba67680e19, State: Initialized, Role: FOLLOWER
I20260619 18:46:21.966101 6706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f44e1a96ce34db9a86e45ba67680e19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f44e1a96ce34db9a86e45ba67680e19" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 38961 } }
I20260619 18:46:21.966248 6706 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f44e1a96ce34db9a86e45ba67680e19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:21.966332 6706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f44e1a96ce34db9a86e45ba67680e19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:21.966450 6706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f44e1a96ce34db9a86e45ba67680e19 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:21.967515 6706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f44e1a96ce34db9a86e45ba67680e19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f44e1a96ce34db9a86e45ba67680e19" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 38961 } }
I20260619 18:46:21.967834 6706 leader_election.cc:302] T 00000000000000000000000000000000 P 5f44e1a96ce34db9a86e45ba67680e19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f44e1a96ce34db9a86e45ba67680e19; no voters:
I20260619 18:46:21.968155 6706 leader_election.cc:288] T 00000000000000000000000000000000 P 5f44e1a96ce34db9a86e45ba67680e19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:21.968233 6711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f44e1a96ce34db9a86e45ba67680e19 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:21.968498 6711 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f44e1a96ce34db9a86e45ba67680e19 [term 1 LEADER]: Becoming Leader. State: Replica: 5f44e1a96ce34db9a86e45ba67680e19, State: Running, Role: LEADER
I20260619 18:46:21.968823 6711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f44e1a96ce34db9a86e45ba67680e19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f44e1a96ce34db9a86e45ba67680e19" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 38961 } }
I20260619 18:46:21.969287 6706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f44e1a96ce34db9a86e45ba67680e19 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:21.970335 6713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f44e1a96ce34db9a86e45ba67680e19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f44e1a96ce34db9a86e45ba67680e19. Latest consensus state: current_term: 1 leader_uuid: "5f44e1a96ce34db9a86e45ba67680e19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f44e1a96ce34db9a86e45ba67680e19" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 38961 } } }
I20260619 18:46:21.970456 6713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f44e1a96ce34db9a86e45ba67680e19 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:21.970846 6720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:21.970808 6712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f44e1a96ce34db9a86e45ba67680e19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f44e1a96ce34db9a86e45ba67680e19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f44e1a96ce34db9a86e45ba67680e19" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 38961 } } }
I20260619 18:46:21.970953 6712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f44e1a96ce34db9a86e45ba67680e19 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:21.974104 6720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:21.979645 6720 catalog_manager.cc:1347] Generated new cluster ID: 0fa8f4963b53404498c897243341f9db
I20260619 18:46:21.979707 6720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:21.994781 6720 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:21.995746 6720 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:22.004760 6720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f44e1a96ce34db9a86e45ba67680e19: Generated new TSK 0
I20260619 18:46:22.005419 6720 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.143.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab.
Entry for principal kudu/127.21.143.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab.
I20260619 18:46:22.035696 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.65:0
--local_ip_for_outbound_sockets=127.21.143.65
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:38961
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:40673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:22.143950 6732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:22.144227 6732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:22.144297 6732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:22.148356 6732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:22.148435 6732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:22.148520 6732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.65
I20260619 18:46:22.153627 6732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:40673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:38961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.65
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:22.154711 6732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:22.155881 6732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:22.165391 6737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:22.165338 6740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:22.165464 6738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:22.165807 6732 server_base.cc:1034] running on GCE node
I20260619 18:46:22.166132 6732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:22.166745 6732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:22.167903 6732 hybrid_clock.cc:648] HybridClock initialized: now 1781894782167885 us; error 28 us; skew 500 ppm
Jun 19 18:46:22 dist-test-slave-mld5 krb5kdc[6625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894782, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:22.170763 6732 init.cc:377] Logged in from keytab as kudu/127.21.143.65@KRBTEST.COM (short username kudu)
I20260619 18:46:22.171765 6732 webserver.cc:460] Webserver started at http://127.21.143.65:43581/ using document root <none> and password file <none>
I20260619 18:46:22.172286 6732 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:22.172340 6732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:22.172501 6732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:22.174465 6732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-0/data/instance:
uuid: "00b53b31c8c3431c9cfbd18b71ad2171"
format_stamp: "Formatted at 2026-06-19 18:46:22 on dist-test-slave-mld5"
server_key: "8e931127b719853e22f574600afd3804"
server_key_iv: "bcdcba1cdfa6ca001a989c07d00ca89e"
server_key_version: "encryptionkey@0"
I20260619 18:46:22.174983 6732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance:
uuid: "00b53b31c8c3431c9cfbd18b71ad2171"
format_stamp: "Formatted at 2026-06-19 18:46:22 on dist-test-slave-mld5"
server_key: "8e931127b719853e22f574600afd3804"
server_key_iv: "bcdcba1cdfa6ca001a989c07d00ca89e"
server_key_version: "encryptionkey@0"
I20260619 18:46:22.178699 6732 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 18:46:22.181411 6747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:22.182706 6732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 18:46:22.182850 6732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-0/wal
uuid: "00b53b31c8c3431c9cfbd18b71ad2171"
format_stamp: "Formatted at 2026-06-19 18:46:22 on dist-test-slave-mld5"
server_key: "8e931127b719853e22f574600afd3804"
server_key_iv: "bcdcba1cdfa6ca001a989c07d00ca89e"
server_key_version: "encryptionkey@0"
I20260619 18:46:22.182971 6732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:22.212004 6732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:22.215514 6732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:22.215727 6732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:22.216358 6732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:46:22.217213 6732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:46:22.217269 6732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:22.217356 6732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:46:22.217396 6732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:22.227023 6732 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.65:33553
I20260619 18:46:22.227103 6860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.65:33553 every 8 connection(s)
I20260619 18:46:22.228252 6732 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
I20260619 18:46:22.231823 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 6732
I20260619 18:46:22.231962 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance
I20260619 18:46:22.232267 22077 external_mini_cluster.cc:1428] Setting key a4b93b0d9d33af1408df5e4a20d7122e
Jun 19 18:46:22 dist-test-slave-mld5 krb5kdc[6625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894782, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.143.66@KRBTEST.COM; defaulting to no policy
I20260619 18:46:22.243835 6709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:22.230100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:44685 (local address 127.21.143.126:38961)
0619 18:46:22.230253 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:22.230256 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:22.231138 (+ 882us) server_negotiation.cc:407] Connection header received
0619 18:46:22.232798 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:22.232802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:22.232859 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:22.232951 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:22.234375 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:22.234969 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:22.236804 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:22.237047 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:22.237129 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:22.240016 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:22.240029 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:22.240032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:22.240064 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:22.241555 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:22.242121 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:22.242125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:22.242126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:22.242207 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:22.242615 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:22.242619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:22.242621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:22.242872 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:22.242977 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:22.243526 (+ 549us) server_negotiation.cc:299] Negotiation successful
0619 18:46:22.243654 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.66@KRBTEST.COM" created.
I20260619 18:46:22.244668 6863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:22.230479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:38961 (local address 127.21.143.65:44685)
0619 18:46:22.230977 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:22.231021 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:22.232537 (+ 1516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:22.233161 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:22.233169 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:22.233705 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:22.234229 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:22.234243 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:22.235136 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:22.235142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:22.236674 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:22.236688 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:22.237202 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:22.237209 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:22.237440 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:22.238095 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:22.238117 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:22.239854 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:22.241715 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:22.241722 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:22.241738 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:22.242012 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:22.242306 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:22.242309 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:22.242311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:22.242507 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:22.242985 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:22.242993 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:22.243241 (+ 248us) client_negotiation.cc:769] Sending connection context
0619 18:46:22.243473 (+ 232us) client_negotiation.cc:240] Negotiation successful
0619 18:46:22.243801 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":111,"threads_started":1}
I20260619 18:46:22.245904 6861 heartbeater.cc:347] Connected to a master server at 127.21.143.126:38961
I20260619 18:46:22.246158 6861 heartbeater.cc:464] Registering TS with master...
I20260619 18:46:22.246657 6861 heartbeater.cc:511] Master 127.21.143.126:38961 requested a full tablet report, sending...
I20260619 18:46:22.248154 6670 ts_manager.cc:194] Registered new tserver with Master: 00b53b31c8c3431c9cfbd18b71ad2171 (127.21.143.65:33553)
I20260619 18:46:22.249260 6670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.65@KRBTEST.COM'} at 127.21.143.65:44685
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab.
Entry for principal kudu/127.21.143.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab.
I20260619 18:46:22.258574 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.66:0
--local_ip_for_outbound_sockets=127.21.143.66
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:38961
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:40673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:22.365998 6866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:22.366225 6866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:22.366266 6866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:22.370427 6866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:22.370482 6866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:22.370539 6866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.66
I20260619 18:46:22.375684 6866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:40673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:38961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.66
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:22.376598 6866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:22.377514 6866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:22.386507 6872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:22.386514 6874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:22.386556 6871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:22.386886 6866 server_base.cc:1034] running on GCE node
I20260619 18:46:22.387292 6866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:22.387950 6866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:22.389146 6866 hybrid_clock.cc:648] HybridClock initialized: now 1781894782389120 us; error 47 us; skew 500 ppm
Jun 19 18:46:22 dist-test-slave-mld5 krb5kdc[6625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894782, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:22.391747 6866 init.cc:377] Logged in from keytab as kudu/127.21.143.66@KRBTEST.COM (short username kudu)
I20260619 18:46:22.392635 6866 webserver.cc:460] Webserver started at http://127.21.143.66:42791/ using document root <none> and password file <none>
I20260619 18:46:22.393179 6866 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:22.393267 6866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:22.393487 6866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:22.395531 6866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-1/data/instance:
uuid: "a3e89ef48c3d41abb28590436de1e62b"
format_stamp: "Formatted at 2026-06-19 18:46:22 on dist-test-slave-mld5"
server_key: "eb10ff86349d7d94d1e49991a979d78e"
server_key_iv: "4c74318cf4a0ee860636739517c964ae"
server_key_version: "encryptionkey@0"
I20260619 18:46:22.396070 6866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance:
uuid: "a3e89ef48c3d41abb28590436de1e62b"
format_stamp: "Formatted at 2026-06-19 18:46:22 on dist-test-slave-mld5"
server_key: "eb10ff86349d7d94d1e49991a979d78e"
server_key_iv: "4c74318cf4a0ee860636739517c964ae"
server_key_version: "encryptionkey@0"
I20260619 18:46:22.399727 6866 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:46:22.402062 6881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:22.403220 6866 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:46:22.403394 6866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-1/wal
uuid: "a3e89ef48c3d41abb28590436de1e62b"
format_stamp: "Formatted at 2026-06-19 18:46:22 on dist-test-slave-mld5"
server_key: "eb10ff86349d7d94d1e49991a979d78e"
server_key_iv: "4c74318cf4a0ee860636739517c964ae"
server_key_version: "encryptionkey@0"
I20260619 18:46:22.403507 6866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:22.416460 6866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:22.419173 6866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:22.419415 6866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:22.419981 6866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:46:22.420797 6866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:46:22.420877 6866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:22.420961 6866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:46:22.421002 6866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:22.429827 6866 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.66:35855
I20260619 18:46:22.429852 6994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.66:35855 every 8 connection(s)
I20260619 18:46:22.430681 6866 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
I20260619 18:46:22.434430 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 6866
I20260619 18:46:22.434582 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance
I20260619 18:46:22.434967 22077 external_mini_cluster.cc:1428] Setting key c13ad5ac1eb757befbceb3bb8353fda4
Jun 19 18:46:22 dist-test-slave-mld5 krb5kdc[6625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894782, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.143.67@KRBTEST.COM; defaulting to no policy
I20260619 18:46:22.445956 6709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:22.432425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:34481 (local address 127.21.143.126:38961)
0619 18:46:22.432563 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:22.432566 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:22.434054 (+ 1488us) server_negotiation.cc:407] Connection header received
0619 18:46:22.435821 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:22.435824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:22.435880 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:22.435985 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:22.437460 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:22.438014 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:22.439169 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:22.439462 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:22.439545 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:22.442404 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:22.442407 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:22.442409 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:22.442430 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:22.443889 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:22.444389 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:22.444395 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:22.444397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:22.444475 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:22.444815 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:22.444819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:22.444821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:22.444994 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:22.445099 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:22.445630 (+ 531us) server_negotiation.cc:299] Negotiation successful
0619 18:46:22.445762 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 18:46:22.446743 6997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:22.432771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:38961 (local address 127.21.143.66:34481)
0619 18:46:22.433904 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:22.433955 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:22.435612 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:22.436159 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:22.436167 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:22.436592 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:22.437307 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:22.437329 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:22.438141 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:22.438145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:22.439024 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:22.439033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:22.439613 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:22.439620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:22.439851 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:22.440457 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:22.440484 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:22.442229 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:22.444005 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:22.444011 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:22.444024 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:22.444279 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:22.444572 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:22.444575 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:22.444577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:22.444713 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:22.445096 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:22.445102 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:22.445344 (+ 242us) client_negotiation.cc:769] Sending connection context
0619 18:46:22.445568 (+ 224us) client_negotiation.cc:240] Negotiation successful
0619 18:46:22.445900 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.67@KRBTEST.COM" created.
I20260619 18:46:22.447959 6995 heartbeater.cc:347] Connected to a master server at 127.21.143.126:38961
I20260619 18:46:22.448200 6995 heartbeater.cc:464] Registering TS with master...
I20260619 18:46:22.448678 6995 heartbeater.cc:511] Master 127.21.143.126:38961 requested a full tablet report, sending...
I20260619 18:46:22.449702 6670 ts_manager.cc:194] Registered new tserver with Master: a3e89ef48c3d41abb28590436de1e62b (127.21.143.66:35855)
I20260619 18:46:22.450253 6670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.66@KRBTEST.COM'} at 127.21.143.66:34481
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab.
Entry for principal kudu/127.21.143.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab.
I20260619 18:46:22.461899 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.67:0
--local_ip_for_outbound_sockets=127.21.143.67
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:38961
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:40673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:22.571943 7000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:22.572237 7000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:22.572342 7000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:22.576256 7000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:22.576330 7000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:22.576436 7000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.67
I20260619 18:46:22.581441 7000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:40673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:38961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.67
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:22.582381 7000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:22.583384 7000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:22.592275 7006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:22.592276 7005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:22.592399 7000 server_base.cc:1034] running on GCE node
W20260619 18:46:22.592276 7008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:22.592950 7000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:22.593510 7000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:22.594691 7000 hybrid_clock.cc:648] HybridClock initialized: now 1781894782594658 us; error 42 us; skew 500 ppm
Jun 19 18:46:22 dist-test-slave-mld5 krb5kdc[6625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894782, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:22.597335 7000 init.cc:377] Logged in from keytab as kudu/127.21.143.67@KRBTEST.COM (short username kudu)
I20260619 18:46:22.598215 7000 webserver.cc:460] Webserver started at http://127.21.143.67:45931/ using document root <none> and password file <none>
I20260619 18:46:22.598737 7000 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:22.598793 7000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:22.599015 7000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:22.600930 7000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-2/data/instance:
uuid: "8b6789d29acd42d680eb36af86a82e16"
format_stamp: "Formatted at 2026-06-19 18:46:22 on dist-test-slave-mld5"
server_key: "4ac83598adc132d980f6a7aae3b6cba9"
server_key_iv: "b75af5dbd42564081205820ba254feda"
server_key_version: "encryptionkey@0"
I20260619 18:46:22.601467 7000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance:
uuid: "8b6789d29acd42d680eb36af86a82e16"
format_stamp: "Formatted at 2026-06-19 18:46:22 on dist-test-slave-mld5"
server_key: "4ac83598adc132d980f6a7aae3b6cba9"
server_key_iv: "b75af5dbd42564081205820ba254feda"
server_key_version: "encryptionkey@0"
I20260619 18:46:22.604879 7000 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:46:22.607075 7015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:22.608373 7000 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 18:46:22.608511 7000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-2/wal
uuid: "8b6789d29acd42d680eb36af86a82e16"
format_stamp: "Formatted at 2026-06-19 18:46:22 on dist-test-slave-mld5"
server_key: "4ac83598adc132d980f6a7aae3b6cba9"
server_key_iv: "b75af5dbd42564081205820ba254feda"
server_key_version: "encryptionkey@0"
I20260619 18:46:22.608615 7000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:22.644088 7000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:22.646912 7000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:22.647114 7000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:22.647782 7000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:46:22.648591 7000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:46:22.648649 7000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:22.648687 7000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:46:22.648702 7000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:22.656855 7000 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.67:41425
I20260619 18:46:22.656905 7128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.67:41425 every 8 connection(s)
I20260619 18:46:22.657807 7000 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
Jun 19 18:46:22 dist-test-slave-mld5 krb5kdc[6625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894782, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:46:22.667831 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 7000
I20260619 18:46:22.667989 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance
I20260619 18:46:22.668270 22077 external_mini_cluster.cc:1428] Setting key 60e21fb287eb18f3aadc8d80c99ce183
I20260619 18:46:22.671110 6709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:22.659796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:53247 (local address 127.21.143.126:38961)
0619 18:46:22.659942 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:22.659945 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:22.660745 (+ 800us) server_negotiation.cc:407] Connection header received
0619 18:46:22.662142 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:22.662145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:22.662193 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:22.662263 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:22.663359 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:22.663893 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:22.665002 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:22.665207 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:22.665281 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:22.667581 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:22.667584 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:22.667586 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:22.667609 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:22.669091 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:22.669585 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:22.669590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:22.669592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:22.669702 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:22.670035 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:22.670040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:22.670041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:22.670257 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:22.670375 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:22.670862 (+ 487us) server_negotiation.cc:299] Negotiation successful
0619 18:46:22.670983 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 18:46:22.671948 7131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:22.660126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:38961 (local address 127.21.143.67:53247)
0619 18:46:22.660586 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:22.660628 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:22.661857 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:22.662424 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:22.662430 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:22.662754 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:22.663200 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:22.663211 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:22.663997 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:22.664000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:22.664896 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:22.664905 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:22.665327 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:22.665329 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:22.665466 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:22.665985 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:22.666000 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:22.667439 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:22.669194 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:22.669199 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:22.669212 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:22.669455 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:22.669789 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:22.669793 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:22.669795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:22.669940 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:22.670376 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:22.670382 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:22.670659 (+ 277us) client_negotiation.cc:769] Sending connection context
0619 18:46:22.670814 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 18:46:22.671110 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":109,"threads_started":1}
I20260619 18:46:22.672938 7129 heartbeater.cc:347] Connected to a master server at 127.21.143.126:38961
I20260619 18:46:22.673175 7129 heartbeater.cc:464] Registering TS with master...
I20260619 18:46:22.673614 7129 heartbeater.cc:511] Master 127.21.143.126:38961 requested a full tablet report, sending...
I20260619 18:46:22.674540 6670 ts_manager.cc:194] Registered new tserver with Master: 8b6789d29acd42d680eb36af86a82e16 (127.21.143.67:41425)
I20260619 18:46:22.675096 6670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.67@KRBTEST.COM'} at 127.21.143.67:53247
I20260619 18:46:22.681860 22077 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:46:22.690762 6709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:22.683788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43796 (local address 127.21.143.126:38961)
0619 18:46:22.683916 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:22.683920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:22.684102 (+ 182us) server_negotiation.cc:407] Connection header received
0619 18:46:22.684243 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:22.684246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:22.684286 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:22.684369 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:22.685042 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:22.685609 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:22.686468 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:22.686696 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:22.686797 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:22.688007 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:22.688010 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:22.688013 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:22.688035 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:22.689153 (+ 1118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:22.689577 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:22.689582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:22.689583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:22.689658 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:22.689922 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:22.689926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:22.689928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:22.690131 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:22.690263 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:22.690500 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 18:46:22.690608 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 18:46:22.693408 6670 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:46:22.696300 6670 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:46:22.710148 7143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:22.704688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:35855 (local address 127.0.0.1:51382)
0619 18:46:22.704983 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:22.704996 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:22.705077 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:22.705913 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:22.705922 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:22.705950 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:46:22.706148 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:22.706156 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:22.707672 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:22.707676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:22.708775 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:22.708784 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:22.709579 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:22.709582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:22.709690 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:22.709723 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 18:46:22.709873 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 18:46:22.709993 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":88,"thread_start_us":88,"threads_started":1}
I20260619 18:46:22.710294 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:22.704099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:41425 (local address 127.0.0.1:51258)
0619 18:46:22.704437 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:22.704478 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:22.704596 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:22.705962 (+ 1366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:22.705965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:22.705977 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:46:22.706155 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:22.706168 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:22.707618 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:22.707621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:22.708789 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:22.708795 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:22.709900 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:22.709904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:22.710022 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:22.710043 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 18:46:22.710102 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 18:46:22.710157 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":92,"thread_start_us":95,"threads_started":1}
I20260619 18:46:22.710371 7141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:22.704250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51258 (local address 127.21.143.67:41425)
0619 18:46:22.705536 (+ 1286us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:22.705549 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:22.705570 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:46:22.705640 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:22.705645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:22.705803 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:22.706108 (+ 305us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:46:22.706243 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:22.707515 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:22.708920 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:22.709745 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:22.709987 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:22.710099 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:22.710138 (+ 39us) server_negotiation.cc:299] Negotiation successful
0619 18:46:22.710222 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1177,"thread_start_us":108,"threads_started":1}
I20260619 18:46:22.710588 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:22.705089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51382 (local address 127.21.143.66:35855)
0619 18:46:22.705417 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:22.705433 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:22.705458 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:46:22.705537 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:22.705542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:22.705740 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:22.705890 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:46:22.706297 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:22.707493 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:22.708897 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:22.709459 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:22.709580 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:22.709715 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:22.710309 (+ 594us) server_negotiation.cc:299] Negotiation successful
0619 18:46:22.710412 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":122,"threads_started":1}
I20260619 18:46:22.711097 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:22.704758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:33553 (local address 127.0.0.1:47964)
0619 18:46:22.705146 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:22.705162 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:22.705255 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:22.706200 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:22.706206 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:22.706227 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:46:22.706346 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:22.706352 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:22.707283 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:22.707288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:22.708782 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:22.708793 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:22.709881 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:22.709885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:22.710745 (+ 860us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:22.710772 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 18:46:22.710875 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 18:46:22.710945 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":45,"thread_start_us":80,"threads_started":1}
I20260619 18:46:22.711601 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:22.705175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47964 (local address 127.21.143.65:33553)
0619 18:46:22.705668 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:22.705681 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:22.705700 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:46:22.705768 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:22.705772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:22.705931 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:22.706062 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:46:22.706458 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:22.707163 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:22.708920 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:22.709745 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:22.709878 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:22.710004 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:22.711348 (+ 1344us) server_negotiation.cc:299] Negotiation successful
0619 18:46:22.711460 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":108,"threads_started":1}
I20260619 18:46:22.713269 6929 tablet_service.cc:1467] Processing CreateTablet for tablet 5e514481d6c0479da6e68fd41270f0ec (DEFAULT_TABLE table=test-table [id=6f54122470ae40cfb59367327f76d479]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:46:22.713757 6795 tablet_service.cc:1467] Processing CreateTablet for tablet 5e514481d6c0479da6e68fd41270f0ec (DEFAULT_TABLE table=test-table [id=6f54122470ae40cfb59367327f76d479]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:46:22.714222 6929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e514481d6c0479da6e68fd41270f0ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:22.714625 6795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e514481d6c0479da6e68fd41270f0ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:22.720757 7146 tablet_bootstrap.cc:492] T 5e514481d6c0479da6e68fd41270f0ec P 00b53b31c8c3431c9cfbd18b71ad2171: Bootstrap starting.
I20260619 18:46:22.721235 7063 tablet_service.cc:1467] Processing CreateTablet for tablet 5e514481d6c0479da6e68fd41270f0ec (DEFAULT_TABLE table=test-table [id=6f54122470ae40cfb59367327f76d479]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:46:22.722070 7063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e514481d6c0479da6e68fd41270f0ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:22.722704 7146 tablet_bootstrap.cc:654] T 5e514481d6c0479da6e68fd41270f0ec P 00b53b31c8c3431c9cfbd18b71ad2171: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:22.720384 7147 tablet_bootstrap.cc:492] T 5e514481d6c0479da6e68fd41270f0ec P a3e89ef48c3d41abb28590436de1e62b: Bootstrap starting.
I20260619 18:46:22.723881 7146 log.cc:824] T 5e514481d6c0479da6e68fd41270f0ec P 00b53b31c8c3431c9cfbd18b71ad2171: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:22.725430 7147 tablet_bootstrap.cc:654] T 5e514481d6c0479da6e68fd41270f0ec P a3e89ef48c3d41abb28590436de1e62b: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:22.726162 7146 tablet_bootstrap.cc:492] T 5e514481d6c0479da6e68fd41270f0ec P 00b53b31c8c3431c9cfbd18b71ad2171: No bootstrap required, opened a new log
I20260619 18:46:22.726375 7146 ts_tablet_manager.cc:1397] T 5e514481d6c0479da6e68fd41270f0ec P 00b53b31c8c3431c9cfbd18b71ad2171: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 18:46:22.726739 7147 log.cc:824] T 5e514481d6c0479da6e68fd41270f0ec P a3e89ef48c3d41abb28590436de1e62b: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:22.728324 7150 tablet_bootstrap.cc:492] T 5e514481d6c0479da6e68fd41270f0ec P 8b6789d29acd42d680eb36af86a82e16: Bootstrap starting.
I20260619 18:46:22.728632 7147 tablet_bootstrap.cc:492] T 5e514481d6c0479da6e68fd41270f0ec P a3e89ef48c3d41abb28590436de1e62b: No bootstrap required, opened a new log
I20260619 18:46:22.728858 7147 ts_tablet_manager.cc:1397] T 5e514481d6c0479da6e68fd41270f0ec P a3e89ef48c3d41abb28590436de1e62b: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260619 18:46:22.730454 7150 tablet_bootstrap.cc:654] T 5e514481d6c0479da6e68fd41270f0ec P 8b6789d29acd42d680eb36af86a82e16: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:22.730502 7146 raft_consensus.cc:348] T 5e514481d6c0479da6e68fd41270f0ec P 00b53b31c8c3431c9cfbd18b71ad2171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b6789d29acd42d680eb36af86a82e16" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 41425 } } peers { permanent_uuid: "00b53b31c8c3431c9cfbd18b71ad2171" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 33553 } } peers { permanent_uuid: "a3e89ef48c3d41abb28590436de1e62b" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35855 } }
I20260619 18:46:22.730767 7146 raft_consensus.cc:374] T 5e514481d6c0479da6e68fd41270f0ec P 00b53b31c8c3431c9cfbd18b71ad2171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:22.730847 7146 raft_consensus.cc:729] T 5e514481d6c0479da6e68fd41270f0ec P 00b53b31c8c3431c9cfbd18b71ad2171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b53b31c8c3431c9cfbd18b71ad2171, State: Initialized, Role: FOLLOWER
I20260619 18:46:22.731283 7146 consensus_queue.cc:260] T 5e514481d6c0479da6e68fd41270f0ec P 00b53b31c8c3431c9cfbd18b71ad2171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b6789d29acd42d680eb36af86a82e16" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 41425 } } peers { permanent_uuid: "00b53b31c8c3431c9cfbd18b71ad2171" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 33553 } } peers { permanent_uuid: "a3e89ef48c3d41abb28590436de1e62b" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35855 } }
I20260619 18:46:22.731948 6861 heartbeater.cc:503] Master 127.21.143.126:38961 was elected leader, sending a full tablet report...
I20260619 18:46:22.732228 7146 ts_tablet_manager.cc:1428] T 5e514481d6c0479da6e68fd41270f0ec P 00b53b31c8c3431c9cfbd18b71ad2171: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260619 18:46:22.732437 7150 log.cc:824] T 5e514481d6c0479da6e68fd41270f0ec P 8b6789d29acd42d680eb36af86a82e16: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:22.733573 7147 raft_consensus.cc:348] T 5e514481d6c0479da6e68fd41270f0ec P a3e89ef48c3d41abb28590436de1e62b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b6789d29acd42d680eb36af86a82e16" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 41425 } } peers { permanent_uuid: "00b53b31c8c3431c9cfbd18b71ad2171" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 33553 } } peers { permanent_uuid: "a3e89ef48c3d41abb28590436de1e62b" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35855 } }
I20260619 18:46:22.733777 7147 raft_consensus.cc:374] T 5e514481d6c0479da6e68fd41270f0ec P a3e89ef48c3d41abb28590436de1e62b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:22.733844 7147 raft_consensus.cc:729] T 5e514481d6c0479da6e68fd41270f0ec P a3e89ef48c3d41abb28590436de1e62b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e89ef48c3d41abb28590436de1e62b, State: Initialized, Role: FOLLOWER
I20260619 18:46:22.734311 7150 tablet_bootstrap.cc:492] T 5e514481d6c0479da6e68fd41270f0ec P 8b6789d29acd42d680eb36af86a82e16: No bootstrap required, opened a new log
I20260619 18:46:22.734272 7147 consensus_queue.cc:260] T 5e514481d6c0479da6e68fd41270f0ec P a3e89ef48c3d41abb28590436de1e62b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b6789d29acd42d680eb36af86a82e16" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 41425 } } peers { permanent_uuid: "00b53b31c8c3431c9cfbd18b71ad2171" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 33553 } } peers { permanent_uuid: "a3e89ef48c3d41abb28590436de1e62b" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35855 } }
I20260619 18:46:22.734547 7150 ts_tablet_manager.cc:1397] T 5e514481d6c0479da6e68fd41270f0ec P 8b6789d29acd42d680eb36af86a82e16: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 18:46:22.735011 6995 heartbeater.cc:503] Master 127.21.143.126:38961 was elected leader, sending a full tablet report...
I20260619 18:46:22.735139 7147 ts_tablet_manager.cc:1428] T 5e514481d6c0479da6e68fd41270f0ec P a3e89ef48c3d41abb28590436de1e62b: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260619 18:46:22.738818 7150 raft_consensus.cc:348] T 5e514481d6c0479da6e68fd41270f0ec P 8b6789d29acd42d680eb36af86a82e16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b6789d29acd42d680eb36af86a82e16" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 41425 } } peers { permanent_uuid: "00b53b31c8c3431c9cfbd18b71ad2171" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 33553 } } peers { permanent_uuid: "a3e89ef48c3d41abb28590436de1e62b" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35855 } }
I20260619 18:46:22.738996 7150 raft_consensus.cc:374] T 5e514481d6c0479da6e68fd41270f0ec P 8b6789d29acd42d680eb36af86a82e16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:22.739068 7150 raft_consensus.cc:729] T 5e514481d6c0479da6e68fd41270f0ec P 8b6789d29acd42d680eb36af86a82e16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b6789d29acd42d680eb36af86a82e16, State: Initialized, Role: FOLLOWER
I20260619 18:46:22.739478 7150 consensus_queue.cc:260] T 5e514481d6c0479da6e68fd41270f0ec P 8b6789d29acd42d680eb36af86a82e16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b6789d29acd42d680eb36af86a82e16" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 41425 } } peers { permanent_uuid: "00b53b31c8c3431c9cfbd18b71ad2171" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 33553 } } peers { permanent_uuid: "a3e89ef48c3d41abb28590436de1e62b" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35855 } }
I20260619 18:46:22.740072 7129 heartbeater.cc:503] Master 127.21.143.126:38961 was elected leader, sending a full tablet report...
I20260619 18:46:22.740330 7150 ts_tablet_manager.cc:1428] T 5e514481d6c0479da6e68fd41270f0ec P 8b6789d29acd42d680eb36af86a82e16: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
W20260619 18:46:22.908743 7130 tablet.cc:2369] T 5e514481d6c0479da6e68fd41270f0ec P 8b6789d29acd42d680eb36af86a82e16: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:46:22.931879 6996 tablet.cc:2369] T 5e514481d6c0479da6e68fd41270f0ec P a3e89ef48c3d41abb28590436de1e62b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:46:22.979615 6862 tablet.cc:2369] T 5e514481d6c0479da6e68fd41270f0ec P 00b53b31c8c3431c9cfbd18b71ad2171: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:46:23.050454 7153 raft_consensus.cc:482] T 5e514481d6c0479da6e68fd41270f0ec P a3e89ef48c3d41abb28590436de1e62b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:46:23.050645 7153 raft_consensus.cc:504] T 5e514481d6c0479da6e68fd41270f0ec P a3e89ef48c3d41abb28590436de1e62b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b6789d29acd42d680eb36af86a82e16" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 41425 } } peers { permanent_uuid: "00b53b31c8c3431c9cfbd18b71ad2171" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 33553 } } peers { permanent_uuid: "a3e89ef48c3d41abb28590436de1e62b" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35855 } }
I20260619 18:46:23.051712 7153 leader_election.cc:288] T 5e514481d6c0479da6e68fd41270f0ec P a3e89ef48c3d41abb28590436de1e62b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b6789d29acd42d680eb36af86a82e16 (127.21.143.67:41425), 00b53b31c8c3431c9cfbd18b71ad2171 (127.21.143.65:33553)
I20260619 18:46:23.055754 7141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:23.052087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:37413 (local address 127.21.143.67:41425)
0619 18:46:23.052226 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:23.052231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:23.052418 (+ 187us) server_negotiation.cc:407] Connection header received
0619 18:46:23.052615 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:23.052618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:23.052670 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:23.052755 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:46:23.053136 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:23.053641 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:23.054702 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:23.055112 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:23.055247 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:23.055342 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:23.055567 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 18:46:23.055633 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 18:46:23.055774 7155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:23.052024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:41425 (local address 127.21.143.66:37413)
0619 18:46:23.052321 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:23.052337 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:23.052499 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:23.052767 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:23.052770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:23.052788 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:46:23.052984 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:23.052990 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:23.053753 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:23.053758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:23.054581 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:23.054589 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:23.055240 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:23.055243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:23.055391 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:23.055458 (+ 67us) client_negotiation.cc:769] Sending connection context
0619 18:46:23.055550 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 18:46:23.055621 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":25,"thread_start_us":64,"threads_started":1}
I20260619 18:46:23.055879 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:23.052518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:40445 (local address 127.21.143.65:33553)
0619 18:46:23.052634 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:23.052639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:23.052730 (+ 91us) server_negotiation.cc:407] Connection header received
0619 18:46:23.052883 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:23.052886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:23.052959 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:23.053057 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:46:23.053328 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:23.053826 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:23.054839 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:23.055216 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:23.055395 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:23.055476 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:23.055689 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 18:46:23.055755 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 18:46:23.055933 7156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:23.052324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:33553 (local address 127.21.143.66:40445)
0619 18:46:23.052633 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:23.052646 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:23.052771 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:23.053066 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:23.053069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:23.053085 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:46:23.053221 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:23.053227 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:23.053940 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:23.053943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:23.054722 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:23.054730 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:23.055390 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:23.055393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:23.055568 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:23.055587 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 18:46:23.055675 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 18:46:23.055743 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":72,"thread_start_us":77,"threads_started":1}
I20260619 18:46:23.056356 7083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e514481d6c0479da6e68fd41270f0ec" candidate_uuid: "a3e89ef48c3d41abb28590436de1e62b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b6789d29acd42d680eb36af86a82e16" is_pre_election: true
I20260619 18:46:23.056421 6815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e514481d6c0479da6e68fd41270f0ec" candidate_uuid: "a3e89ef48c3d41abb28590436de1e62b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b53b31c8c3431c9cfbd18b71ad2171" is_pre_election: true
I20260619 18:46:23.056664 7083 raft_consensus.cc:2454] T 5e514481d6c0479da6e68fd41270f0ec P 8b6789d29acd42d680eb36af86a82e16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3e89ef48c3d41abb28590436de1e62b in term 0.
I20260619 18:46:23.056704 6815 raft_consensus.cc:2454] T 5e514481d6c0479da6e68fd41270f0ec P 00b53b31c8c3431c9cfbd18b71ad2171 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3e89ef48c3d41abb28590436de1e62b in term 0.
I20260619 18:46:23.057173 6883 leader_election.cc:302] T 5e514481d6c0479da6e68fd41270f0ec P a3e89ef48c3d41abb28590436de1e62b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b53b31c8c3431c9cfbd18b71ad2171, a3e89ef48c3d41abb28590436de1e62b; no voters:
I20260619 18:46:23.057411 7153 raft_consensus.cc:2781] T 5e514481d6c0479da6e68fd41270f0ec P a3e89ef48c3d41abb28590436de1e62b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:46:23.057513 7153 raft_consensus.cc:482] T 5e514481d6c0479da6e68fd41270f0ec P a3e89ef48c3d41abb28590436de1e62b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:46:23.057559 7153 raft_consensus.cc:3037] T 5e514481d6c0479da6e68fd41270f0ec P a3e89ef48c3d41abb28590436de1e62b [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:23.058596 7153 raft_consensus.cc:504] T 5e514481d6c0479da6e68fd41270f0ec P a3e89ef48c3d41abb28590436de1e62b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b6789d29acd42d680eb36af86a82e16" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 41425 } } peers { permanent_uuid: "00b53b31c8c3431c9cfbd18b71ad2171" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 33553 } } peers { permanent_uuid: "a3e89ef48c3d41abb28590436de1e62b" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35855 } }
I20260619 18:46:23.058915 7153 leader_election.cc:288] T 5e514481d6c0479da6e68fd41270f0ec P a3e89ef48c3d41abb28590436de1e62b [CANDIDATE]: Term 1 election: Requested vote from peers 8b6789d29acd42d680eb36af86a82e16 (127.21.143.67:41425), 00b53b31c8c3431c9cfbd18b71ad2171 (127.21.143.65:33553)
I20260619 18:46:23.059242 7083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e514481d6c0479da6e68fd41270f0ec" candidate_uuid: "a3e89ef48c3d41abb28590436de1e62b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b6789d29acd42d680eb36af86a82e16"
I20260619 18:46:23.059325 6815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e514481d6c0479da6e68fd41270f0ec" candidate_uuid: "a3e89ef48c3d41abb28590436de1e62b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b53b31c8c3431c9cfbd18b71ad2171"
I20260619 18:46:23.059450 7083 raft_consensus.cc:3037] T 5e514481d6c0479da6e68fd41270f0ec P 8b6789d29acd42d680eb36af86a82e16 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:23.059465 6815 raft_consensus.cc:3037] T 5e514481d6c0479da6e68fd41270f0ec P 00b53b31c8c3431c9cfbd18b71ad2171 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:23.060417 7083 raft_consensus.cc:2454] T 5e514481d6c0479da6e68fd41270f0ec P 8b6789d29acd42d680eb36af86a82e16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3e89ef48c3d41abb28590436de1e62b in term 1.
I20260619 18:46:23.060465 6815 raft_consensus.cc:2454] T 5e514481d6c0479da6e68fd41270f0ec P 00b53b31c8c3431c9cfbd18b71ad2171 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3e89ef48c3d41abb28590436de1e62b in term 1.
I20260619 18:46:23.060724 6883 leader_election.cc:302] T 5e514481d6c0479da6e68fd41270f0ec P a3e89ef48c3d41abb28590436de1e62b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b53b31c8c3431c9cfbd18b71ad2171, a3e89ef48c3d41abb28590436de1e62b; no voters:
I20260619 18:46:23.060917 7153 raft_consensus.cc:2781] T 5e514481d6c0479da6e68fd41270f0ec P a3e89ef48c3d41abb28590436de1e62b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:23.061151 7153 raft_consensus.cc:686] T 5e514481d6c0479da6e68fd41270f0ec P a3e89ef48c3d41abb28590436de1e62b [term 1 LEADER]: Becoming Leader. State: Replica: a3e89ef48c3d41abb28590436de1e62b, State: Running, Role: LEADER
I20260619 18:46:23.061496 7153 consensus_queue.cc:237] T 5e514481d6c0479da6e68fd41270f0ec P a3e89ef48c3d41abb28590436de1e62b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b6789d29acd42d680eb36af86a82e16" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 41425 } } peers { permanent_uuid: "00b53b31c8c3431c9cfbd18b71ad2171" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 33553 } } peers { permanent_uuid: "a3e89ef48c3d41abb28590436de1e62b" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35855 } }
I20260619 18:46:23.064359 6669 catalog_manager.cc:5495] T 5e514481d6c0479da6e68fd41270f0ec P a3e89ef48c3d41abb28590436de1e62b reported cstate change: term changed from 0 to 1, leader changed from <none> to a3e89ef48c3d41abb28590436de1e62b (127.21.143.66). New cstate: current_term: 1 leader_uuid: "a3e89ef48c3d41abb28590436de1e62b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b6789d29acd42d680eb36af86a82e16" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 41425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b53b31c8c3431c9cfbd18b71ad2171" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 33553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3e89ef48c3d41abb28590436de1e62b" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35855 } health_report { overall_health: HEALTHY } } }
I20260619 18:46:23.157537 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:23.154005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51392 (local address 127.21.143.66:35855)
0619 18:46:23.154148 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:23.154153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:23.154168 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:46:23.154268 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:23.154271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:23.154326 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:23.154439 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:46:23.154694 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:23.155190 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:23.156111 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:23.156329 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:23.156438 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:23.156749 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:46:23.157149 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:46:23.157281 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:23.157325 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 18:46:23.157378 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 18:46:23.163789 6815 raft_consensus.cc:1261] T 5e514481d6c0479da6e68fd41270f0ec P 00b53b31c8c3431c9cfbd18b71ad2171 [term 1 FOLLOWER]: Refusing update from remote peer a3e89ef48c3d41abb28590436de1e62b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:46:23.163789 7083 raft_consensus.cc:1261] T 5e514481d6c0479da6e68fd41270f0ec P 8b6789d29acd42d680eb36af86a82e16 [term 1 FOLLOWER]: Refusing update from remote peer a3e89ef48c3d41abb28590436de1e62b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:46:23.164482 7153 consensus_queue.cc:1035] T 5e514481d6c0479da6e68fd41270f0ec P a3e89ef48c3d41abb28590436de1e62b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b6789d29acd42d680eb36af86a82e16" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 41425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:46:23.164710 7157 consensus_queue.cc:1035] T 5e514481d6c0479da6e68fd41270f0ec P a3e89ef48c3d41abb28590436de1e62b [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b53b31c8c3431c9cfbd18b71ad2171" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 33553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:46:23.179551 6669 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:46:23.179693 6669 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:46:23.181841 6669 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5f44e1a96ce34db9a86e45ba67680e19: Sending DeleteTablet for 3 replicas of tablet 5e514481d6c0479da6e68fd41270f0ec
I20260619 18:46:23.182583 7063 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e514481d6c0479da6e68fd41270f0ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:46:23 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:51258
I20260619 18:46:23.182654 6795 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e514481d6c0479da6e68fd41270f0ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:46:23 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:47964
I20260619 18:46:23.183133 7170 tablet_replica.cc:330] T 5e514481d6c0479da6e68fd41270f0ec P 00b53b31c8c3431c9cfbd18b71ad2171: stopping tablet replica
I20260619 18:46:23.183087 6929 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e514481d6c0479da6e68fd41270f0ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:46:23 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:51382
I20260619 18:46:23.183454 7170 raft_consensus.cc:2229] T 5e514481d6c0479da6e68fd41270f0ec P 00b53b31c8c3431c9cfbd18b71ad2171 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:46:23.183501 7171 tablet_replica.cc:330] T 5e514481d6c0479da6e68fd41270f0ec P a3e89ef48c3d41abb28590436de1e62b: stopping tablet replica
I20260619 18:46:23.183789 7169 tablet_replica.cc:330] T 5e514481d6c0479da6e68fd41270f0ec P 8b6789d29acd42d680eb36af86a82e16: stopping tablet replica
I20260619 18:46:23.184072 7169 raft_consensus.cc:2229] T 5e514481d6c0479da6e68fd41270f0ec P 8b6789d29acd42d680eb36af86a82e16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:46:23.184262 7171 raft_consensus.cc:2229] T 5e514481d6c0479da6e68fd41270f0ec P a3e89ef48c3d41abb28590436de1e62b [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:46:23.184368 7169 raft_consensus.cc:2258] T 5e514481d6c0479da6e68fd41270f0ec P 8b6789d29acd42d680eb36af86a82e16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:46:23.184403 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 6732
I20260619 18:46:23.184635 7171 raft_consensus.cc:2258] T 5e514481d6c0479da6e68fd41270f0ec P a3e89ef48c3d41abb28590436de1e62b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:46:23.186077 7171 ts_tablet_manager.cc:1905] T 5e514481d6c0479da6e68fd41270f0ec P a3e89ef48c3d41abb28590436de1e62b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:46:23.188726 7169 ts_tablet_manager.cc:1905] T 5e514481d6c0479da6e68fd41270f0ec P 8b6789d29acd42d680eb36af86a82e16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:46:23.188988 7171 ts_tablet_manager.cc:1918] T 5e514481d6c0479da6e68fd41270f0ec P a3e89ef48c3d41abb28590436de1e62b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:46:23.189144 7171 log.cc:1196] T 5e514481d6c0479da6e68fd41270f0ec P a3e89ef48c3d41abb28590436de1e62b: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-1/wal/wals/5e514481d6c0479da6e68fd41270f0ec
I20260619 18:46:23.189548 7171 ts_tablet_manager.cc:1939] T 5e514481d6c0479da6e68fd41270f0ec P a3e89ef48c3d41abb28590436de1e62b: Deleting consensus metadata
I20260619 18:46:23.190748 6654 catalog_manager.cc:4841] TS a3e89ef48c3d41abb28590436de1e62b (127.21.143.66:35855): tablet 5e514481d6c0479da6e68fd41270f0ec (table test-table [id=6f54122470ae40cfb59367327f76d479]) successfully deleted
I20260619 18:46:23.192236 7169 ts_tablet_manager.cc:1918] T 5e514481d6c0479da6e68fd41270f0ec P 8b6789d29acd42d680eb36af86a82e16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260619 18:46:23.192251 6655 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.143.65:33553 (error 108)
I20260619 18:46:23.192391 7169 log.cc:1196] T 5e514481d6c0479da6e68fd41270f0ec P 8b6789d29acd42d680eb36af86a82e16: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894542572984-22077-0/minicluster-data/ts-2/wal/wals/5e514481d6c0479da6e68fd41270f0ec
I20260619 18:46:23.192696 7169 ts_tablet_manager.cc:1939] T 5e514481d6c0479da6e68fd41270f0ec P 8b6789d29acd42d680eb36af86a82e16: Deleting consensus metadata
W20260619 18:46:23.193256 7142 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:23.192963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:33553 (local address 127.0.0.1:47970)
0619 18:46:23.193094 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:23.193164 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.65:33553: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 18:46:23.193392 6655 catalog_manager.cc:4584] TS 00b53b31c8c3431c9cfbd18b71ad2171 (127.21.143.65:33553): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5e514481d6c0479da6e68fd41270f0ec: Network error: Client connection negotiation failed: client connection to 127.21.143.65:33553: connect: Connection reset by peer (error 104)
I20260619 18:46:23.193712 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 6866
I20260619 18:46:23.195251 6655 catalog_manager.cc:4841] TS 8b6789d29acd42d680eb36af86a82e16 (127.21.143.67:41425): tablet 5e514481d6c0479da6e68fd41270f0ec (table test-table [id=6f54122470ae40cfb59367327f76d479]) successfully deleted
I20260619 18:46:23.199390 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 7000
I20260619 18:46:23.204596 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 6638
2026-06-19T18:46:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2591 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T18:46:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:23Z Disabled control of system clock
I20260619 18:46:23.234994 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:37271
--webserver_interface=127.21.143.126
--webserver_port=0
--builtin_ntp_servers=127.21.143.84:35007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:37271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 18:46:23.363229 7176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:23.363698 7176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:23.363755 7176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:23.367714 7176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:23.367766 7176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:23.367781 7176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:23.367798 7176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:23.367813 7176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:23.372470 7176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:35007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:37271
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:37271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:23.373478 7176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:23.374799 7176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:23.383144 7181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:23.383170 7182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:23.383267 7176 server_base.cc:1034] running on GCE node
W20260619 18:46:23.383190 7184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:23.383898 7176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:23.384932 7176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:23.386088 7176 hybrid_clock.cc:648] HybridClock initialized: now 1781894783386071 us; error 29 us; skew 500 ppm
I20260619 18:46:23.387832 7176 webserver.cc:460] Webserver started at http://127.21.143.126:42937/ using document root <none> and password file <none>
I20260619 18:46:23.388384 7176 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:23.388450 7176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:23.388677 7176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:23.390668 7176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "d3c08d62837d4bf1baacacaa71324833"
format_stamp: "Formatted at 2026-06-19 18:46:23 on dist-test-slave-mld5"
server_key: "9e244ca9ec6ef3e4d0d5b5264af423fc"
server_key_iv: "e5ccb1128583301bdebfe5cd5d5038ce"
server_key_version: "encryptionkey@0"
I20260619 18:46:23.391208 7176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "d3c08d62837d4bf1baacacaa71324833"
format_stamp: "Formatted at 2026-06-19 18:46:23 on dist-test-slave-mld5"
server_key: "9e244ca9ec6ef3e4d0d5b5264af423fc"
server_key_iv: "e5ccb1128583301bdebfe5cd5d5038ce"
server_key_version: "encryptionkey@0"
I20260619 18:46:23.394666 7176 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 18:46:23.396996 7190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:23.398160 7176 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260619 18:46:23.398288 7176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "d3c08d62837d4bf1baacacaa71324833"
format_stamp: "Formatted at 2026-06-19 18:46:23 on dist-test-slave-mld5"
server_key: "9e244ca9ec6ef3e4d0d5b5264af423fc"
server_key_iv: "e5ccb1128583301bdebfe5cd5d5038ce"
server_key_version: "encryptionkey@0"
I20260619 18:46:23.398402 7176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:23.412202 7176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:23.412863 7176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:23.413051 7176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:23.419867 7176 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:37271
I20260619 18:46:23.419881 7242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:37271 every 8 connection(s)
I20260619 18:46:23.420782 7176 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:23.423751 7243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:23.429188 7243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3c08d62837d4bf1baacacaa71324833: Bootstrap starting.
I20260619 18:46:23.430675 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 7176
I20260619 18:46:23.430809 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:46:23.431092 22077 external_mini_cluster.cc:1428] Setting key b40e6683c644d9cefaff9f0c60de09d6
I20260619 18:46:23.431871 7243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3c08d62837d4bf1baacacaa71324833: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:23.432723 7243 log.cc:824] T 00000000000000000000000000000000 P d3c08d62837d4bf1baacacaa71324833: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:23.434728 7243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3c08d62837d4bf1baacacaa71324833: No bootstrap required, opened a new log
I20260619 18:46:23.438920 7243 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3c08d62837d4bf1baacacaa71324833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c08d62837d4bf1baacacaa71324833" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37271 } }
I20260619 18:46:23.439210 7243 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3c08d62837d4bf1baacacaa71324833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:23.439283 7243 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3c08d62837d4bf1baacacaa71324833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3c08d62837d4bf1baacacaa71324833, State: Initialized, Role: FOLLOWER
I20260619 18:46:23.439805 7243 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3c08d62837d4bf1baacacaa71324833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c08d62837d4bf1baacacaa71324833" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37271 } }
I20260619 18:46:23.439932 7243 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3c08d62837d4bf1baacacaa71324833 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:23.440014 7243 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3c08d62837d4bf1baacacaa71324833 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:23.440112 7243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3c08d62837d4bf1baacacaa71324833 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:23.440178 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:23.432204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46448 (local address 127.21.143.126:37271)
0619 18:46:23.432793 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:23.432808 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:23.432864 (+ 56us) server_negotiation.cc:407] Connection header received
0619 18:46:23.433661 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:23.433678 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:23.433997 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:23.434258 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:23.434575 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:23.435641 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:23.436835 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:23.437307 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:23.437425 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:23.438296 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:23.438301 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:23.438312 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:23.438673 (+ 361us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:23.438726 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:23.438745 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:23.438817 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:23.438981 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 18:46:23.439258 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":137,"threads_started":1}
I20260619 18:46:23.440969 7243 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3c08d62837d4bf1baacacaa71324833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c08d62837d4bf1baacacaa71324833" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37271 } }
I20260619 18:46:23.441323 7243 leader_election.cc:302] T 00000000000000000000000000000000 P d3c08d62837d4bf1baacacaa71324833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3c08d62837d4bf1baacacaa71324833; no voters:
I20260619 18:46:23.441640 7243 leader_election.cc:288] T 00000000000000000000000000000000 P d3c08d62837d4bf1baacacaa71324833 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:23.441746 7248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3c08d62837d4bf1baacacaa71324833 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:23.441985 7248 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3c08d62837d4bf1baacacaa71324833 [term 1 LEADER]: Becoming Leader. State: Replica: d3c08d62837d4bf1baacacaa71324833, State: Running, Role: LEADER
I20260619 18:46:23.442549 7248 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3c08d62837d4bf1baacacaa71324833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c08d62837d4bf1baacacaa71324833" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37271 } }
I20260619 18:46:23.442845 7243 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3c08d62837d4bf1baacacaa71324833 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:23.444204 7249 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c08d62837d4bf1baacacaa71324833 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3c08d62837d4bf1baacacaa71324833" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c08d62837d4bf1baacacaa71324833" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37271 } } }
I20260619 18:46:23.444344 7249 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c08d62837d4bf1baacacaa71324833 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:23.444774 7250 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c08d62837d4bf1baacacaa71324833 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3c08d62837d4bf1baacacaa71324833. Latest consensus state: current_term: 1 leader_uuid: "d3c08d62837d4bf1baacacaa71324833" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c08d62837d4bf1baacacaa71324833" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37271 } } }
I20260619 18:46:23.445082 7250 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c08d62837d4bf1baacacaa71324833 [sys.catalog]: This master's current role is: LEADER
W20260619 18:46:23.447621 7263 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d3c08d62837d4bf1baacacaa71324833: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:46:23.447721 7263 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:46:23.447803 7260 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:23.448549 7260 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:23.453552 7260 catalog_manager.cc:1347] Generated new cluster ID: 8d4bfdc34ce0429ba00062e08ff74e3a
I20260619 18:46:23.453610 7260 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:23.471041 7260 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:23.472222 7260 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:23.481734 7260 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3c08d62837d4bf1baacacaa71324833: Generated new TSK 0
I20260619 18:46:23.482470 7260 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:23.495771 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.65:0
--local_ip_for_outbound_sockets=127.21.143.65
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:37271
--builtin_ntp_servers=127.21.143.84:35007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 18:46:23.605468 7267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:23.605705 7267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:23.605767 7267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:23.609754 7267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:23.609802 7267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:23.609853 7267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.65
I20260619 18:46:23.614496 7267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:35007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:37271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.65
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:23.615566 7267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:23.616931 7267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:23.626503 7275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:23.626521 7272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:23.626519 7273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:23.626906 7267 server_base.cc:1034] running on GCE node
I20260619 18:46:23.627362 7267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:23.627977 7267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:23.629133 7267 hybrid_clock.cc:648] HybridClock initialized: now 1781894783629122 us; error 22 us; skew 500 ppm
I20260619 18:46:23.630836 7267 webserver.cc:460] Webserver started at http://127.21.143.65:37627/ using document root <none> and password file <none>
I20260619 18:46:23.631464 7267 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:23.631529 7267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:23.631772 7267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:23.633764 7267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-0/data/instance:
uuid: "7463777c88c746d4a7adae243527f2fc"
format_stamp: "Formatted at 2026-06-19 18:46:23 on dist-test-slave-mld5"
server_key: "5fb694ff71934c15d22ced95b34edeeb"
server_key_iv: "721fbd0b854297ffdcbb44fe2f2269e5"
server_key_version: "encryptionkey@0"
I20260619 18:46:23.634327 7267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance:
uuid: "7463777c88c746d4a7adae243527f2fc"
format_stamp: "Formatted at 2026-06-19 18:46:23 on dist-test-slave-mld5"
server_key: "5fb694ff71934c15d22ced95b34edeeb"
server_key_iv: "721fbd0b854297ffdcbb44fe2f2269e5"
server_key_version: "encryptionkey@0"
I20260619 18:46:23.637910 7267 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 18:46:23.640192 7281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:23.641352 7267 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:46:23.641481 7267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-0/wal
uuid: "7463777c88c746d4a7adae243527f2fc"
format_stamp: "Formatted at 2026-06-19 18:46:23 on dist-test-slave-mld5"
server_key: "5fb694ff71934c15d22ced95b34edeeb"
server_key_iv: "721fbd0b854297ffdcbb44fe2f2269e5"
server_key_version: "encryptionkey@0"
I20260619 18:46:23.641590 7267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:23.667559 7267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:23.668288 7267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:23.668500 7267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:23.669107 7267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:46:23.670070 7267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:46:23.670142 7267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:23.670217 7267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:46:23.670259 7267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:23.679478 7267 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.65:36485
I20260619 18:46:23.679507 7394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.65:36485 every 8 connection(s)
I20260619 18:46:23.680389 7267 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
I20260619 18:46:23.681885 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 7267
I20260619 18:46:23.681975 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance
I20260619 18:46:23.682334 22077 external_mini_cluster.cc:1428] Setting key 759cbed55bb9663ff806c7bf9964f4c1
I20260619 18:46:23.684195 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.66:0
--local_ip_for_outbound_sockets=127.21.143.66
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:37271
--builtin_ntp_servers=127.21.143.84:35007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 18:46:23.690904 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:23.682234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:48959 (local address 127.21.143.126:37271)
0619 18:46:23.682400 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:23.682406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:23.683594 (+ 1188us) server_negotiation.cc:407] Connection header received
0619 18:46:23.685104 (+ 1510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:23.685107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:23.685160 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:23.685238 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:23.686451 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:23.686969 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:23.688836 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:23.689050 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:23.689129 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:23.690446 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:23.690449 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:23.690451 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:23.690462 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:23.690481 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:23.690488 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:23.690544 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:23.690743 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 18:46:23.690793 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 18:46:23.692210 7397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:23.682699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:37271 (local address 127.21.143.65:48959)
0619 18:46:23.683441 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:23.683494 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:23.684897 (+ 1403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:23.685434 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:23.685446 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:23.685571 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:23.686287 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:23.686307 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:23.687097 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:23.687104 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:23.688710 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:23.688725 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:23.689208 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:23.689214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:23.689467 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:23.690124 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:46:23.690129 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:23.690190 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:46:23.690284 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:23.690575 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:23.690587 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:23.690597 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 18:46:23.690733 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 18:46:23.691058 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":124,"threads_started":1}
I20260619 18:46:23.693502 7395 heartbeater.cc:347] Connected to a master server at 127.21.143.126:37271
I20260619 18:46:23.693759 7395 heartbeater.cc:464] Registering TS with master...
I20260619 18:46:23.694254 7395 heartbeater.cc:511] Master 127.21.143.126:37271 requested a full tablet report, sending...
I20260619 18:46:23.695914 7207 ts_manager.cc:194] Registered new tserver with Master: 7463777c88c746d4a7adae243527f2fc (127.21.143.65:36485)
I20260619 18:46:23.697430 7207 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.143.65:48959
W20260619 18:46:23.795509 7398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:23.795763 7398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:23.795809 7398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:23.799763 7398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:23.799860 7398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:23.799947 7398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.66
I20260619 18:46:23.804816 7398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:35007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:37271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.66
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:23.805858 7398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:23.807188 7398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:23.816496 7406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:23.816504 7404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:23.816504 7403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:23.816849 7398 server_base.cc:1034] running on GCE node
I20260619 18:46:23.817289 7398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:23.817900 7398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:23.819083 7398 hybrid_clock.cc:648] HybridClock initialized: now 1781894783819069 us; error 25 us; skew 500 ppm
I20260619 18:46:23.820853 7398 webserver.cc:460] Webserver started at http://127.21.143.66:43571/ using document root <none> and password file <none>
I20260619 18:46:23.821426 7398 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:23.821514 7398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:23.821719 7398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:23.823979 7398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-1/data/instance:
uuid: "465577fbc37449a8bd7f923d203f4e0d"
format_stamp: "Formatted at 2026-06-19 18:46:23 on dist-test-slave-mld5"
server_key: "986ef803a8df724a850a709387d5752f"
server_key_iv: "45afbca2f23638ddbec83e7f56c1c928"
server_key_version: "encryptionkey@0"
I20260619 18:46:23.824525 7398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance:
uuid: "465577fbc37449a8bd7f923d203f4e0d"
format_stamp: "Formatted at 2026-06-19 18:46:23 on dist-test-slave-mld5"
server_key: "986ef803a8df724a850a709387d5752f"
server_key_iv: "45afbca2f23638ddbec83e7f56c1c928"
server_key_version: "encryptionkey@0"
I20260619 18:46:23.828162 7398 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:46:23.830420 7412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:23.831671 7398 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 18:46:23.831802 7398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-1/wal
uuid: "465577fbc37449a8bd7f923d203f4e0d"
format_stamp: "Formatted at 2026-06-19 18:46:23 on dist-test-slave-mld5"
server_key: "986ef803a8df724a850a709387d5752f"
server_key_iv: "45afbca2f23638ddbec83e7f56c1c928"
server_key_version: "encryptionkey@0"
I20260619 18:46:23.831914 7398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:23.857543 7398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:23.858300 7398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:23.858513 7398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:23.859149 7398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:46:23.860158 7398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:46:23.860229 7398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:23.860297 7398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:46:23.860350 7398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:23.869074 7398 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.66:39919
I20260619 18:46:23.869109 7525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.66:39919 every 8 connection(s)
I20260619 18:46:23.869987 7398 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
I20260619 18:46:23.871085 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 7398
I20260619 18:46:23.871220 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance
I20260619 18:46:23.871552 22077 external_mini_cluster.cc:1428] Setting key b244d22982f55860af205ab9adff5f05
I20260619 18:46:23.873633 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.67:0
--local_ip_for_outbound_sockets=127.21.143.67
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:37271
--builtin_ntp_servers=127.21.143.84:35007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 18:46:23.880196 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:23.871812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:47613 (local address 127.21.143.126:37271)
0619 18:46:23.871959 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:23.871964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:23.873206 (+ 1242us) server_negotiation.cc:407] Connection header received
0619 18:46:23.874646 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:23.874652 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:23.874717 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:23.874812 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:23.876042 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:23.876906 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:23.877999 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:23.878429 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:23.878555 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:23.879615 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:23.879621 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:23.879627 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:23.879646 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:23.879674 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:23.879687 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:23.879768 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:23.879920 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 18:46:23.879993 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 18:46:23.880975 7528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:23.872284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:37271 (local address 127.21.143.66:47613)
0619 18:46:23.873045 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:23.873099 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:23.874447 (+ 1348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:23.875027 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:23.875038 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:23.875163 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:23.875897 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:23.875916 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:23.877022 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:23.877024 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:23.877880 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:23.877889 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:23.878568 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:23.878571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:23.878710 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:23.879367 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:46:23.879368 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:23.879407 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:46:23.879466 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:23.879789 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:23.879795 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:23.879799 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 18:46:23.879886 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 18:46:23.880102 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":99,"threads_started":1}
I20260619 18:46:23.882129 7526 heartbeater.cc:347] Connected to a master server at 127.21.143.126:37271
I20260619 18:46:23.882387 7526 heartbeater.cc:464] Registering TS with master...
I20260619 18:46:23.883046 7526 heartbeater.cc:511] Master 127.21.143.126:37271 requested a full tablet report, sending...
I20260619 18:46:23.884222 7207 ts_manager.cc:194] Registered new tserver with Master: 465577fbc37449a8bd7f923d203f4e0d (127.21.143.66:39919)
I20260619 18:46:23.885066 7207 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.143.66:47613
W20260619 18:46:23.987237 7529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:23.987537 7529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:23.987573 7529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:23.991554 7529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:23.991602 7529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:23.991654 7529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.67
I20260619 18:46:23.996217 7529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:35007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:37271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.67
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:23.997157 7529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:23.998456 7529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:24.007578 7537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:24.007555 7535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:24.007550 7534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:24.007788 7529 server_base.cc:1034] running on GCE node
I20260619 18:46:24.008215 7529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:24.008831 7529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:24.010015 7529 hybrid_clock.cc:648] HybridClock initialized: now 1781894784010005 us; error 30 us; skew 500 ppm
I20260619 18:46:24.011780 7529 webserver.cc:460] Webserver started at http://127.21.143.67:33119/ using document root <none> and password file <none>
I20260619 18:46:24.012326 7529 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:24.012413 7529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:24.012656 7529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:24.014644 7529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-2/data/instance:
uuid: "288a627bb4854f2fbe607f68b63cca2f"
format_stamp: "Formatted at 2026-06-19 18:46:24 on dist-test-slave-mld5"
server_key: "9bfc56221ccdadb94317569503c6a4e6"
server_key_iv: "1c57b6a72b0822a60cf4f511fee51876"
server_key_version: "encryptionkey@0"
I20260619 18:46:24.015240 7529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance:
uuid: "288a627bb4854f2fbe607f68b63cca2f"
format_stamp: "Formatted at 2026-06-19 18:46:24 on dist-test-slave-mld5"
server_key: "9bfc56221ccdadb94317569503c6a4e6"
server_key_iv: "1c57b6a72b0822a60cf4f511fee51876"
server_key_version: "encryptionkey@0"
I20260619 18:46:24.018747 7529 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:46:24.021116 7543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:24.022236 7529 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:46:24.022374 7529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-2/wal
uuid: "288a627bb4854f2fbe607f68b63cca2f"
format_stamp: "Formatted at 2026-06-19 18:46:24 on dist-test-slave-mld5"
server_key: "9bfc56221ccdadb94317569503c6a4e6"
server_key_iv: "1c57b6a72b0822a60cf4f511fee51876"
server_key_version: "encryptionkey@0"
I20260619 18:46:24.022475 7529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:24.050153 7529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:24.050900 7529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:24.051111 7529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:24.051780 7529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:46:24.052803 7529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:46:24.052874 7529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:24.052946 7529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:46:24.052999 7529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:24.062008 7529 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.67:44277
I20260619 18:46:24.062044 7656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.67:44277 every 8 connection(s)
I20260619 18:46:24.062897 7529 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
I20260619 18:46:24.070444 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 7529
I20260619 18:46:24.070569 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance
I20260619 18:46:24.070863 22077 external_mini_cluster.cc:1428] Setting key b1d67c0836e78793693d7cbf29ec8ecc
I20260619 18:46:24.072455 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:24.064794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:42529 (local address 127.21.143.126:37271)
0619 18:46:24.064935 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:24.064939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:24.066028 (+ 1089us) server_negotiation.cc:407] Connection header received
0619 18:46:24.067445 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:24.067449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:24.067507 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:24.067614 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:24.068506 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:24.069044 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:24.070185 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:24.070511 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:24.070629 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:24.071965 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:24.071968 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:24.071971 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:24.071983 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:24.072011 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:24.072019 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:24.072094 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:24.072263 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 18:46:24.072321 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 18:46:24.073338 7659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:24.065264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:37271 (local address 127.21.143.67:42529)
0619 18:46:24.065880 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:24.065919 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:24.067206 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:24.067789 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:24.067796 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:24.067880 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:24.068362 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:24.068374 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:24.069174 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:24.069182 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:24.070037 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:24.070045 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:24.070725 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:24.070730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:24.070922 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:24.071694 (+ 772us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:46:24.071696 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:24.071735 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:46:24.071795 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:24.072109 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:24.072119 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:24.072129 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 18:46:24.072249 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 18:46:24.072536 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":105,"threads_started":1}
I20260619 18:46:24.074335 7657 heartbeater.cc:347] Connected to a master server at 127.21.143.126:37271
I20260619 18:46:24.074556 7657 heartbeater.cc:464] Registering TS with master...
I20260619 18:46:24.075052 7657 heartbeater.cc:511] Master 127.21.143.126:37271 requested a full tablet report, sending...
I20260619 18:46:24.076086 7207 ts_manager.cc:194] Registered new tserver with Master: 288a627bb4854f2fbe607f68b63cca2f (127.21.143.67:44277)
I20260619 18:46:24.076653 7207 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.143.67:42529
I20260619 18:46:24.084671 22077 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:46:24.090273 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:24.086624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46462 (local address 127.21.143.126:37271)
0619 18:46:24.086760 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:24.086765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:24.086959 (+ 194us) server_negotiation.cc:407] Connection header received
0619 18:46:24.087061 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:24.087064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:24.087105 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:24.087172 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:24.087510 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:24.088050 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:24.088851 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:24.089061 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:24.089140 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:24.089852 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:24.089855 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:24.089857 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:24.089866 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:24.089882 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:24.089888 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:24.089939 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:24.090024 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 18:46:24.090075 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 18:46:24.093019 7207 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46462:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:46:24.096083 7207 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:46:24.110321 7672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:24.105468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32916 (local address 127.21.143.65:36485)
0619 18:46:24.105816 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:24.105830 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:24.105856 (+ 26us) server_negotiation.cc:407] Connection header received
0619 18:46:24.105939 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:24.105943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:24.106153 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:24.106330 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:46:24.106604 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:24.107643 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:24.109084 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:24.109606 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:24.109729 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:24.109928 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:24.110028 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 18:46:24.110141 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":116,"threads_started":1}
I20260619 18:46:24.110301 7670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:24.104732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:36485 (local address 127.0.0.1:32916)
0619 18:46:24.105560 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:24.105575 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:24.105662 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:24.106320 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:24.106326 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:24.106350 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:46:24.106480 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:24.106486 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:24.107756 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:24.107762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:24.108967 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:24.108976 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:24.109725 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:24.109728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:24.109851 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:24.109887 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 18:46:24.110000 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 18:46:24.110134 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"thread_start_us":85,"threads_started":1}
I20260619 18:46:24.111114 7668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:24.104653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:39919 (local address 127.0.0.1:60686)
0619 18:46:24.105021 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:24.105073 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:24.105197 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:24.106876 (+ 1679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:24.106879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:24.106896 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:46:24.106984 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:24.106989 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:24.108365 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:24.108367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:24.109660 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:24.109670 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:24.110719 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:24.110722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:24.110810 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:24.110842 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 18:46:24.110928 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 18:46:24.110991 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":105,"thread_start_us":78,"threads_started":1}
I20260619 18:46:24.111557 7669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:24.105022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:44277 (local address 127.0.0.1:53382)
0619 18:46:24.105841 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:24.105853 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:24.105935 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:24.106683 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:24.106686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:24.106701 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:46:24.106782 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:24.106787 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:24.108311 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:24.108317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:24.109660 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:24.109670 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:24.111165 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:24.111169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:24.111255 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:24.111287 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 18:46:24.111367 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 18:46:24.111446 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"mutex_wait_us":81,"thread_start_us":111,"threads_started":1}
I20260619 18:46:24.112377 7673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:24.105857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53382 (local address 127.21.143.67:44277)
0619 18:46:24.106274 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:24.106287 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:24.106308 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:46:24.106375 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:24.106379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:24.106570 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:24.106711 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:46:24.106874 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:24.108059 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:24.109799 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:24.111033 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:24.111950 (+ 917us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:24.112109 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:24.112155 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 18:46:24.112236 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":184,"threads_started":1}
I20260619 18:46:24.113538 7329 tablet_service.cc:1467] Processing CreateTablet for tablet 56c49654de39478f92185f84fde54b59 (DEFAULT_TABLE table=test-table [id=29c1732858e1460394104e7b93a92552]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:46:24.114696 7329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56c49654de39478f92185f84fde54b59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:24.111251 7671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:24.105353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60686 (local address 127.21.143.66:39919)
0619 18:46:24.106279 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:24.106290 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:24.106311 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:46:24.106381 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:24.106385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:24.106583 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:24.106726 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:46:24.107117 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:24.108255 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:24.109799 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:24.110574 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:24.110694 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:24.110862 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:24.110956 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 18:46:24.111056 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":791,"thread_start_us":113,"threads_started":1}
I20260619 18:46:24.115000 7591 tablet_service.cc:1467] Processing CreateTablet for tablet 56c49654de39478f92185f84fde54b59 (DEFAULT_TABLE table=test-table [id=29c1732858e1460394104e7b93a92552]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:46:24.116021 7591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56c49654de39478f92185f84fde54b59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:24.117445 7460 tablet_service.cc:1467] Processing CreateTablet for tablet 56c49654de39478f92185f84fde54b59 (DEFAULT_TABLE table=test-table [id=29c1732858e1460394104e7b93a92552]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:46:24.118249 7460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56c49654de39478f92185f84fde54b59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:24.122416 7674 tablet_bootstrap.cc:492] T 56c49654de39478f92185f84fde54b59 P 7463777c88c746d4a7adae243527f2fc: Bootstrap starting.
I20260619 18:46:24.123772 7675 tablet_bootstrap.cc:492] T 56c49654de39478f92185f84fde54b59 P 288a627bb4854f2fbe607f68b63cca2f: Bootstrap starting.
I20260619 18:46:24.124913 7674 tablet_bootstrap.cc:654] T 56c49654de39478f92185f84fde54b59 P 7463777c88c746d4a7adae243527f2fc: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:24.125744 7675 tablet_bootstrap.cc:654] T 56c49654de39478f92185f84fde54b59 P 288a627bb4854f2fbe607f68b63cca2f: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:24.125718 7674 log.cc:824] T 56c49654de39478f92185f84fde54b59 P 7463777c88c746d4a7adae243527f2fc: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:24.126531 7675 log.cc:824] T 56c49654de39478f92185f84fde54b59 P 288a627bb4854f2fbe607f68b63cca2f: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:24.127017 7676 tablet_bootstrap.cc:492] T 56c49654de39478f92185f84fde54b59 P 465577fbc37449a8bd7f923d203f4e0d: Bootstrap starting.
I20260619 18:46:24.128500 7675 tablet_bootstrap.cc:492] T 56c49654de39478f92185f84fde54b59 P 288a627bb4854f2fbe607f68b63cca2f: No bootstrap required, opened a new log
I20260619 18:46:24.128500 7674 tablet_bootstrap.cc:492] T 56c49654de39478f92185f84fde54b59 P 7463777c88c746d4a7adae243527f2fc: No bootstrap required, opened a new log
I20260619 18:46:24.128696 7674 ts_tablet_manager.cc:1397] T 56c49654de39478f92185f84fde54b59 P 7463777c88c746d4a7adae243527f2fc: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 18:46:24.128696 7675 ts_tablet_manager.cc:1397] T 56c49654de39478f92185f84fde54b59 P 288a627bb4854f2fbe607f68b63cca2f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 18:46:24.133136 7676 tablet_bootstrap.cc:654] T 56c49654de39478f92185f84fde54b59 P 465577fbc37449a8bd7f923d203f4e0d: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:24.133113 7674 raft_consensus.cc:348] T 56c49654de39478f92185f84fde54b59 P 7463777c88c746d4a7adae243527f2fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465577fbc37449a8bd7f923d203f4e0d" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39919 } } peers { permanent_uuid: "7463777c88c746d4a7adae243527f2fc" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36485 } } peers { permanent_uuid: "288a627bb4854f2fbe607f68b63cca2f" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 44277 } }
I20260619 18:46:24.133443 7674 raft_consensus.cc:374] T 56c49654de39478f92185f84fde54b59 P 7463777c88c746d4a7adae243527f2fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:24.133793 7674 raft_consensus.cc:729] T 56c49654de39478f92185f84fde54b59 P 7463777c88c746d4a7adae243527f2fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7463777c88c746d4a7adae243527f2fc, State: Initialized, Role: FOLLOWER
I20260619 18:46:24.133685 7675 raft_consensus.cc:348] T 56c49654de39478f92185f84fde54b59 P 288a627bb4854f2fbe607f68b63cca2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465577fbc37449a8bd7f923d203f4e0d" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39919 } } peers { permanent_uuid: "7463777c88c746d4a7adae243527f2fc" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36485 } } peers { permanent_uuid: "288a627bb4854f2fbe607f68b63cca2f" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 44277 } }
I20260619 18:46:24.133953 7675 raft_consensus.cc:374] T 56c49654de39478f92185f84fde54b59 P 288a627bb4854f2fbe607f68b63cca2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:24.133998 7675 raft_consensus.cc:729] T 56c49654de39478f92185f84fde54b59 P 288a627bb4854f2fbe607f68b63cca2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 288a627bb4854f2fbe607f68b63cca2f, State: Initialized, Role: FOLLOWER
I20260619 18:46:24.134100 7676 log.cc:824] T 56c49654de39478f92185f84fde54b59 P 465577fbc37449a8bd7f923d203f4e0d: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:24.134334 7674 consensus_queue.cc:260] T 56c49654de39478f92185f84fde54b59 P 7463777c88c746d4a7adae243527f2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465577fbc37449a8bd7f923d203f4e0d" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39919 } } peers { permanent_uuid: "7463777c88c746d4a7adae243527f2fc" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36485 } } peers { permanent_uuid: "288a627bb4854f2fbe607f68b63cca2f" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 44277 } }
I20260619 18:46:24.134349 7675 consensus_queue.cc:260] T 56c49654de39478f92185f84fde54b59 P 288a627bb4854f2fbe607f68b63cca2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465577fbc37449a8bd7f923d203f4e0d" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39919 } } peers { permanent_uuid: "7463777c88c746d4a7adae243527f2fc" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36485 } } peers { permanent_uuid: "288a627bb4854f2fbe607f68b63cca2f" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 44277 } }
I20260619 18:46:24.135035 7395 heartbeater.cc:503] Master 127.21.143.126:37271 was elected leader, sending a full tablet report...
I20260619 18:46:24.135368 7657 heartbeater.cc:503] Master 127.21.143.126:37271 was elected leader, sending a full tablet report...
I20260619 18:46:24.135422 7675 ts_tablet_manager.cc:1428] T 56c49654de39478f92185f84fde54b59 P 288a627bb4854f2fbe607f68b63cca2f: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 18:46:24.136624 7676 tablet_bootstrap.cc:492] T 56c49654de39478f92185f84fde54b59 P 465577fbc37449a8bd7f923d203f4e0d: No bootstrap required, opened a new log
I20260619 18:46:24.136850 7676 ts_tablet_manager.cc:1397] T 56c49654de39478f92185f84fde54b59 P 465577fbc37449a8bd7f923d203f4e0d: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260619 18:46:24.137459 7674 ts_tablet_manager.cc:1428] T 56c49654de39478f92185f84fde54b59 P 7463777c88c746d4a7adae243527f2fc: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260619 18:46:24.141131 7676 raft_consensus.cc:348] T 56c49654de39478f92185f84fde54b59 P 465577fbc37449a8bd7f923d203f4e0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465577fbc37449a8bd7f923d203f4e0d" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39919 } } peers { permanent_uuid: "7463777c88c746d4a7adae243527f2fc" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36485 } } peers { permanent_uuid: "288a627bb4854f2fbe607f68b63cca2f" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 44277 } }
I20260619 18:46:24.141377 7676 raft_consensus.cc:374] T 56c49654de39478f92185f84fde54b59 P 465577fbc37449a8bd7f923d203f4e0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:24.141443 7676 raft_consensus.cc:729] T 56c49654de39478f92185f84fde54b59 P 465577fbc37449a8bd7f923d203f4e0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 465577fbc37449a8bd7f923d203f4e0d, State: Initialized, Role: FOLLOWER
I20260619 18:46:24.141934 7676 consensus_queue.cc:260] T 56c49654de39478f92185f84fde54b59 P 465577fbc37449a8bd7f923d203f4e0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465577fbc37449a8bd7f923d203f4e0d" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39919 } } peers { permanent_uuid: "7463777c88c746d4a7adae243527f2fc" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36485 } } peers { permanent_uuid: "288a627bb4854f2fbe607f68b63cca2f" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 44277 } }
I20260619 18:46:24.142699 7526 heartbeater.cc:503] Master 127.21.143.126:37271 was elected leader, sending a full tablet report...
I20260619 18:46:24.143038 7676 ts_tablet_manager.cc:1428] T 56c49654de39478f92185f84fde54b59 P 465577fbc37449a8bd7f923d203f4e0d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260619 18:46:24.181660 7396 tablet.cc:2369] T 56c49654de39478f92185f84fde54b59 P 7463777c88c746d4a7adae243527f2fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:46:24.313939 7658 tablet.cc:2369] T 56c49654de39478f92185f84fde54b59 P 288a627bb4854f2fbe607f68b63cca2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:46:24.354921 7682 raft_consensus.cc:482] T 56c49654de39478f92185f84fde54b59 P 465577fbc37449a8bd7f923d203f4e0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:46:24.355216 7682 raft_consensus.cc:504] T 56c49654de39478f92185f84fde54b59 P 465577fbc37449a8bd7f923d203f4e0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465577fbc37449a8bd7f923d203f4e0d" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39919 } } peers { permanent_uuid: "7463777c88c746d4a7adae243527f2fc" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36485 } } peers { permanent_uuid: "288a627bb4854f2fbe607f68b63cca2f" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 44277 } }
I20260619 18:46:24.356456 7682 leader_election.cc:288] T 56c49654de39478f92185f84fde54b59 P 465577fbc37449a8bd7f923d203f4e0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7463777c88c746d4a7adae243527f2fc (127.21.143.65:36485), 288a627bb4854f2fbe607f68b63cca2f (127.21.143.67:44277)
I20260619 18:46:24.360078 7528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:24.356667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:36485 (local address 127.21.143.66:59321)
0619 18:46:24.356782 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:24.356801 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:24.356898 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:24.357205 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:24.357208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:24.357227 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:46:24.357346 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:24.357353 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:24.358187 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:24.358192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:24.358972 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:24.358981 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:24.359670 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:24.359674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:24.359771 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:24.359793 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 18:46:24.359879 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 18:46:24.359948 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 18:46:24.360078 7672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:24.356775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:59321 (local address 127.21.143.65:36485)
0619 18:46:24.356922 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:24.356927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:24.356946 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:46:24.357015 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:24.357019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:24.357104 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:24.357189 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:46:24.357478 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:24.357999 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:24.359108 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:24.359542 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:24.359670 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:24.359722 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:24.359894 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 18:46:24.359962 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 18:46:24.360271 7673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:24.356966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:60013 (local address 127.21.143.67:44277)
0619 18:46:24.357124 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:24.357130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:24.357262 (+ 132us) server_negotiation.cc:407] Connection header received
0619 18:46:24.357405 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:24.357408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:24.357469 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:24.357552 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:46:24.357829 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:24.358336 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:24.359369 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:24.359758 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:24.359884 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:24.359927 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:24.360096 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 18:46:24.360160 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 18:46:24.360288 7683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:24.356769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:44277 (local address 127.21.143.66:60013)
0619 18:46:24.357175 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:24.357186 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:24.357296 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:24.357564 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:24.357567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:24.357582 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:46:24.357723 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:24.357729 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:24.358451 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:24.358455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:24.359219 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:24.359226 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:24.359878 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:24.359882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:24.359973 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:24.359998 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 18:46:24.360081 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 18:46:24.360152 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":101,"threads_started":1}
I20260619 18:46:24.360772 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c49654de39478f92185f84fde54b59" candidate_uuid: "465577fbc37449a8bd7f923d203f4e0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7463777c88c746d4a7adae243527f2fc" is_pre_election: true
I20260619 18:46:24.360890 7611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c49654de39478f92185f84fde54b59" candidate_uuid: "465577fbc37449a8bd7f923d203f4e0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288a627bb4854f2fbe607f68b63cca2f" is_pre_election: true
I20260619 18:46:24.361101 7349 raft_consensus.cc:2454] T 56c49654de39478f92185f84fde54b59 P 7463777c88c746d4a7adae243527f2fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 465577fbc37449a8bd7f923d203f4e0d in term 0.
I20260619 18:46:24.361112 7611 raft_consensus.cc:2454] T 56c49654de39478f92185f84fde54b59 P 288a627bb4854f2fbe607f68b63cca2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 465577fbc37449a8bd7f923d203f4e0d in term 0.
I20260619 18:46:24.361621 7415 leader_election.cc:302] T 56c49654de39478f92185f84fde54b59 P 465577fbc37449a8bd7f923d203f4e0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 288a627bb4854f2fbe607f68b63cca2f, 465577fbc37449a8bd7f923d203f4e0d, 7463777c88c746d4a7adae243527f2fc; no voters:
I20260619 18:46:24.361891 7682 raft_consensus.cc:2781] T 56c49654de39478f92185f84fde54b59 P 465577fbc37449a8bd7f923d203f4e0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:46:24.361986 7682 raft_consensus.cc:482] T 56c49654de39478f92185f84fde54b59 P 465577fbc37449a8bd7f923d203f4e0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:46:24.362046 7682 raft_consensus.cc:3037] T 56c49654de39478f92185f84fde54b59 P 465577fbc37449a8bd7f923d203f4e0d [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:24.363106 7682 raft_consensus.cc:504] T 56c49654de39478f92185f84fde54b59 P 465577fbc37449a8bd7f923d203f4e0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465577fbc37449a8bd7f923d203f4e0d" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39919 } } peers { permanent_uuid: "7463777c88c746d4a7adae243527f2fc" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36485 } } peers { permanent_uuid: "288a627bb4854f2fbe607f68b63cca2f" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 44277 } }
I20260619 18:46:24.363495 7682 leader_election.cc:288] T 56c49654de39478f92185f84fde54b59 P 465577fbc37449a8bd7f923d203f4e0d [CANDIDATE]: Term 1 election: Requested vote from peers 7463777c88c746d4a7adae243527f2fc (127.21.143.65:36485), 288a627bb4854f2fbe607f68b63cca2f (127.21.143.67:44277)
I20260619 18:46:24.363852 7349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c49654de39478f92185f84fde54b59" candidate_uuid: "465577fbc37449a8bd7f923d203f4e0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7463777c88c746d4a7adae243527f2fc"
I20260619 18:46:24.363883 7611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c49654de39478f92185f84fde54b59" candidate_uuid: "465577fbc37449a8bd7f923d203f4e0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288a627bb4854f2fbe607f68b63cca2f"
I20260619 18:46:24.363966 7349 raft_consensus.cc:3037] T 56c49654de39478f92185f84fde54b59 P 7463777c88c746d4a7adae243527f2fc [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:24.364009 7611 raft_consensus.cc:3037] T 56c49654de39478f92185f84fde54b59 P 288a627bb4854f2fbe607f68b63cca2f [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:24.364998 7611 raft_consensus.cc:2454] T 56c49654de39478f92185f84fde54b59 P 288a627bb4854f2fbe607f68b63cca2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 465577fbc37449a8bd7f923d203f4e0d in term 1.
I20260619 18:46:24.365005 7349 raft_consensus.cc:2454] T 56c49654de39478f92185f84fde54b59 P 7463777c88c746d4a7adae243527f2fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 465577fbc37449a8bd7f923d203f4e0d in term 1.
I20260619 18:46:24.365330 7415 leader_election.cc:302] T 56c49654de39478f92185f84fde54b59 P 465577fbc37449a8bd7f923d203f4e0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 465577fbc37449a8bd7f923d203f4e0d, 7463777c88c746d4a7adae243527f2fc; no voters:
I20260619 18:46:24.365545 7682 raft_consensus.cc:2781] T 56c49654de39478f92185f84fde54b59 P 465577fbc37449a8bd7f923d203f4e0d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:24.365780 7682 raft_consensus.cc:686] T 56c49654de39478f92185f84fde54b59 P 465577fbc37449a8bd7f923d203f4e0d [term 1 LEADER]: Becoming Leader. State: Replica: 465577fbc37449a8bd7f923d203f4e0d, State: Running, Role: LEADER
I20260619 18:46:24.366103 7682 consensus_queue.cc:237] T 56c49654de39478f92185f84fde54b59 P 465577fbc37449a8bd7f923d203f4e0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465577fbc37449a8bd7f923d203f4e0d" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39919 } } peers { permanent_uuid: "7463777c88c746d4a7adae243527f2fc" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36485 } } peers { permanent_uuid: "288a627bb4854f2fbe607f68b63cca2f" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 44277 } }
I20260619 18:46:24.369176 7206 catalog_manager.cc:5495] T 56c49654de39478f92185f84fde54b59 P 465577fbc37449a8bd7f923d203f4e0d reported cstate change: term changed from 0 to 1, leader changed from <none> to 465577fbc37449a8bd7f923d203f4e0d (127.21.143.66). New cstate: current_term: 1 leader_uuid: "465577fbc37449a8bd7f923d203f4e0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465577fbc37449a8bd7f923d203f4e0d" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 39919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7463777c88c746d4a7adae243527f2fc" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "288a627bb4854f2fbe607f68b63cca2f" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 44277 } health_report { overall_health: UNKNOWN } } }
W20260619 18:46:24.371071 7527 tablet.cc:2369] T 56c49654de39478f92185f84fde54b59 P 465577fbc37449a8bd7f923d203f4e0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:46:24.408308 7671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:24.403641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60696 (local address 127.21.143.66:39919)
0619 18:46:24.403812 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:24.403816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:24.403830 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:46:24.404313 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:24.404319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:24.404407 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:24.404528 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:46:24.404924 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:24.405512 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:24.406389 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:24.406834 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:24.406957 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:24.407218 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:46:24.407827 (+ 609us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:46:24.407960 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:24.408009 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 18:46:24.408082 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260619 18:46:24.415184 7611 raft_consensus.cc:1261] T 56c49654de39478f92185f84fde54b59 P 288a627bb4854f2fbe607f68b63cca2f [term 1 FOLLOWER]: Refusing update from remote peer 465577fbc37449a8bd7f923d203f4e0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:46:24.415184 7349 raft_consensus.cc:1261] T 56c49654de39478f92185f84fde54b59 P 7463777c88c746d4a7adae243527f2fc [term 1 FOLLOWER]: Refusing update from remote peer 465577fbc37449a8bd7f923d203f4e0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:46:24.416004 7682 consensus_queue.cc:1035] T 56c49654de39478f92185f84fde54b59 P 465577fbc37449a8bd7f923d203f4e0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7463777c88c746d4a7adae243527f2fc" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:46:24.416218 7684 consensus_queue.cc:1035] T 56c49654de39478f92185f84fde54b59 P 465577fbc37449a8bd7f923d203f4e0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "288a627bb4854f2fbe607f68b63cca2f" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 44277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:46:24.425586 7689 mvcc.cc:205] Tried to move back new op lower bound from 7298641036956049408 to 7298641036766486528. Current Snapshot: MvccSnapshot[applied={T|T < 7298641036956049408}]
I20260619 18:46:24.435216 7207 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:46:24.435447 7207 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:46:24.438498 7207 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3c08d62837d4bf1baacacaa71324833: Sending DeleteTablet for 3 replicas of tablet 56c49654de39478f92185f84fde54b59
I20260619 18:46:24.439136 7688 mvcc.cc:205] Tried to move back new op lower bound from 7298641036956049408 to 7298641036766486528. Current Snapshot: MvccSnapshot[applied={T|T < 7298641036956049408}]
I20260619 18:46:24.439430 7460 tablet_service.cc:1514] Processing DeleteTablet for tablet 56c49654de39478f92185f84fde54b59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:46:24 UTC) from {username='slave'} at 127.0.0.1:60686
I20260619 18:46:24.439842 7329 tablet_service.cc:1514] Processing DeleteTablet for tablet 56c49654de39478f92185f84fde54b59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:46:24 UTC) from {username='slave'} at 127.0.0.1:32916
I20260619 18:46:24.440505 7591 tablet_service.cc:1514] Processing DeleteTablet for tablet 56c49654de39478f92185f84fde54b59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:46:24 UTC) from {username='slave'} at 127.0.0.1:53382
I20260619 18:46:24.441738 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 7267
I20260619 18:46:24.443322 7697 tablet_replica.cc:330] T 56c49654de39478f92185f84fde54b59 P 465577fbc37449a8bd7f923d203f4e0d: stopping tablet replica
I20260619 18:46:24.443668 7697 raft_consensus.cc:2229] T 56c49654de39478f92185f84fde54b59 P 465577fbc37449a8bd7f923d203f4e0d [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:46:24.444084 7697 raft_consensus.cc:2258] T 56c49654de39478f92185f84fde54b59 P 465577fbc37449a8bd7f923d203f4e0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:46:24.444799 7699 tablet_replica.cc:330] T 56c49654de39478f92185f84fde54b59 P 288a627bb4854f2fbe607f68b63cca2f: stopping tablet replica
I20260619 18:46:24.445088 7699 raft_consensus.cc:2229] T 56c49654de39478f92185f84fde54b59 P 288a627bb4854f2fbe607f68b63cca2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:46:24.445393 7699 raft_consensus.cc:2258] T 56c49654de39478f92185f84fde54b59 P 288a627bb4854f2fbe607f68b63cca2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:46:24.445626 7697 ts_tablet_manager.cc:1905] T 56c49654de39478f92185f84fde54b59 P 465577fbc37449a8bd7f923d203f4e0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:46:24.446628 7699 ts_tablet_manager.cc:1905] T 56c49654de39478f92185f84fde54b59 P 288a627bb4854f2fbe607f68b63cca2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:46:24.448988 7697 ts_tablet_manager.cc:1918] T 56c49654de39478f92185f84fde54b59 P 465577fbc37449a8bd7f923d203f4e0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:46:24.449141 7697 log.cc:1196] T 56c49654de39478f92185f84fde54b59 P 465577fbc37449a8bd7f923d203f4e0d: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-1/wal/wals/56c49654de39478f92185f84fde54b59
I20260619 18:46:24.449291 7699 ts_tablet_manager.cc:1918] T 56c49654de39478f92185f84fde54b59 P 288a627bb4854f2fbe607f68b63cca2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:46:24.449386 7699 log.cc:1196] T 56c49654de39478f92185f84fde54b59 P 288a627bb4854f2fbe607f68b63cca2f: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894542572984-22077-0/minicluster-data/ts-2/wal/wals/56c49654de39478f92185f84fde54b59
I20260619 18:46:24.449651 7697 ts_tablet_manager.cc:1939] T 56c49654de39478f92185f84fde54b59 P 465577fbc37449a8bd7f923d203f4e0d: Deleting consensus metadata
I20260619 18:46:24.449677 7699 ts_tablet_manager.cc:1939] T 56c49654de39478f92185f84fde54b59 P 288a627bb4854f2fbe607f68b63cca2f: Deleting consensus metadata
W20260619 18:46:24.450817 7193 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.143.65:36485 (error 108)
I20260619 18:46:24.450899 7191 catalog_manager.cc:4841] TS 288a627bb4854f2fbe607f68b63cca2f (127.21.143.67:44277): tablet 56c49654de39478f92185f84fde54b59 (table test-table [id=29c1732858e1460394104e7b93a92552]) successfully deleted
I20260619 18:46:24.451045 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 7398
I20260619 18:46:24.452986 7669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:24.452666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:36485 (local address 127.0.0.1:32930)
0619 18:46:24.452802 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:24.452881 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.65:36485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 18:46:24.453191 7193 catalog_manager.cc:4584] TS 7463777c88c746d4a7adae243527f2fc (127.21.143.65:36485): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 56c49654de39478f92185f84fde54b59: Network error: Client connection negotiation failed: client connection to 127.21.143.65:36485: connect: Connection refused (error 111)
I20260619 18:46:24.457162 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 7529
I20260619 18:46:24.458123 7669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:24.457242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:39919 (local address 127.0.0.1:60698)
0619 18:46:24.457965 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:24.458028 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.66:39919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":629}
I20260619 18:46:24.462661 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 7176
2026-06-19T18:46:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1257 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:24 dist-test-slave-mld5 krb5kdc[7703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:24 dist-test-slave-mld5 krb5kdc[7703](info): set up 2 sockets
Jun 19 18:46:24 dist-test-slave-mld5 krb5kdc[7703](info): commencing operation
krb5kdc: starting...
W20260619 18:46:25.495199 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:46:25 dist-test-slave-mld5 krb5kdc[7703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.143.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
I20260619 18:46:25.611925 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:34717
--webserver_interface=127.21.143.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:36105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:34717
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:25.719954 7716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:25.720247 7716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:25.720312 7716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:25.724126 7716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:25.724207 7716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:25.724247 7716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:25.724283 7716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:25.724318 7716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:25.729423 7716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:36105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:34717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:34717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:25.730408 7716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:25.731447 7716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:25.739611 7722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:25.739647 7724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:25.739696 7721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:25.739878 7716 server_base.cc:1034] running on GCE node
I20260619 18:46:25.740478 7716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:25.741534 7716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:25.742717 7716 hybrid_clock.cc:648] HybridClock initialized: now 1781894785742686 us; error 47 us; skew 500 ppm
Jun 19 18:46:25 dist-test-slave-mld5 krb5kdc[7703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894785, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:25.749550 7716 init.cc:377] Logged in from keytab as kudu/127.21.143.126@KRBTEST.COM (short username kudu)
I20260619 18:46:25.750423 7716 webserver.cc:460] Webserver started at http://127.21.143.126:45847/ using document root <none> and password file <none>
I20260619 18:46:25.750957 7716 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:25.751044 7716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:25.751271 7716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:25.753302 7716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "98be9a8be5f44d06a38564be07aab827"
format_stamp: "Formatted at 2026-06-19 18:46:25 on dist-test-slave-mld5"
server_key: "533e7e2553eb8a30af1e4d8ee47fd667"
server_key_iv: "b1345a91975f8f0f3354d780cc21b8ef"
server_key_version: "encryptionkey@0"
I20260619 18:46:25.753928 7716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "98be9a8be5f44d06a38564be07aab827"
format_stamp: "Formatted at 2026-06-19 18:46:25 on dist-test-slave-mld5"
server_key: "533e7e2553eb8a30af1e4d8ee47fd667"
server_key_iv: "b1345a91975f8f0f3354d780cc21b8ef"
server_key_version: "encryptionkey@0"
I20260619 18:46:25.758884 7716 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260619 18:46:25.761668 7731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:25.762820 7716 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:46:25.762966 7716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "98be9a8be5f44d06a38564be07aab827"
format_stamp: "Formatted at 2026-06-19 18:46:25 on dist-test-slave-mld5"
server_key: "533e7e2553eb8a30af1e4d8ee47fd667"
server_key_iv: "b1345a91975f8f0f3354d780cc21b8ef"
server_key_version: "encryptionkey@0"
I20260619 18:46:25.763077 7716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:25.787915 7716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:25.790786 7716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:25.790992 7716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:25.798004 7716 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:34717
I20260619 18:46:25.798055 7783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:34717 every 8 connection(s)
I20260619 18:46:25.798975 7716 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:25.801679 7784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:25.807260 7784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98be9a8be5f44d06a38564be07aab827: Bootstrap starting.
I20260619 18:46:25.807941 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 7716
I20260619 18:46:25.808104 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:46:25.808377 22077 external_mini_cluster.cc:1428] Setting key 7914540f79c1a01a853467a4ce55fc4d
I20260619 18:46:25.810110 7784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98be9a8be5f44d06a38564be07aab827: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:25.811005 7784 log.cc:824] T 00000000000000000000000000000000 P 98be9a8be5f44d06a38564be07aab827: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:25.813263 7784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98be9a8be5f44d06a38564be07aab827: No bootstrap required, opened a new log
I20260619 18:46:25.817294 7784 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98be9a8be5f44d06a38564be07aab827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98be9a8be5f44d06a38564be07aab827" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 34717 } }
I20260619 18:46:25.817595 7784 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98be9a8be5f44d06a38564be07aab827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:25.817659 7784 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98be9a8be5f44d06a38564be07aab827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98be9a8be5f44d06a38564be07aab827, State: Initialized, Role: FOLLOWER
Jun 19 18:46:25 dist-test-slave-mld5 krb5kdc[7703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:46:25.818163 7784 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98be9a8be5f44d06a38564be07aab827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98be9a8be5f44d06a38564be07aab827" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 34717 } }
I20260619 18:46:25.818339 7784 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98be9a8be5f44d06a38564be07aab827 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:25.818416 7784 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98be9a8be5f44d06a38564be07aab827 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:25.818532 7784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98be9a8be5f44d06a38564be07aab827 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:25.819485 7784 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98be9a8be5f44d06a38564be07aab827 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98be9a8be5f44d06a38564be07aab827" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 34717 } }
I20260619 18:46:25.819839 7784 leader_election.cc:302] T 00000000000000000000000000000000 P 98be9a8be5f44d06a38564be07aab827 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98be9a8be5f44d06a38564be07aab827; no voters:
I20260619 18:46:25.820217 7784 leader_election.cc:288] T 00000000000000000000000000000000 P 98be9a8be5f44d06a38564be07aab827 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:25.820335 7789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98be9a8be5f44d06a38564be07aab827 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:25.820556 7789 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98be9a8be5f44d06a38564be07aab827 [term 1 LEADER]: Becoming Leader. State: Replica: 98be9a8be5f44d06a38564be07aab827, State: Running, Role: LEADER
I20260619 18:46:25.820884 7789 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98be9a8be5f44d06a38564be07aab827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98be9a8be5f44d06a38564be07aab827" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 34717 } }
I20260619 18:46:25.821615 7784 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98be9a8be5f44d06a38564be07aab827 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:25.822160 7789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98be9a8be5f44d06a38564be07aab827 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98be9a8be5f44d06a38564be07aab827. Latest consensus state: current_term: 1 leader_uuid: "98be9a8be5f44d06a38564be07aab827" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98be9a8be5f44d06a38564be07aab827" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 34717 } } }
I20260619 18:46:25.822284 7789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98be9a8be5f44d06a38564be07aab827 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:25.822950 7790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98be9a8be5f44d06a38564be07aab827 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98be9a8be5f44d06a38564be07aab827" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98be9a8be5f44d06a38564be07aab827" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 34717 } } }
I20260619 18:46:25.823048 7790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98be9a8be5f44d06a38564be07aab827 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:25.824257 7795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:25.825445 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:25.809682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34250 (local address 127.21.143.126:34717)
0619 18:46:25.810151 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:25.810173 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:25.810217 (+ 44us) server_negotiation.cc:407] Connection header received
0619 18:46:25.811120 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:25.811145 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:25.811594 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:25.811965 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:25.812637 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:25.814095 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:25.814959 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:25.815707 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:25.815897 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:25.818518 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:25.818523 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:25.818537 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:25.818565 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:25.820771 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:25.821324 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:25.821330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:25.821336 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:25.821478 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:25.821816 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:25.821819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:25.821821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:25.822357 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:25.823314 (+ 957us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:25.824008 (+ 694us) server_negotiation.cc:299] Negotiation successful
0619 18:46:25.824369 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":111,"threads_started":1}
I20260619 18:46:25.827531 7795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:25.833097 7795 catalog_manager.cc:1347] Generated new cluster ID: ca2d91e9390a4993b68f64ef806a3f40
I20260619 18:46:25.833156 7795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:25.848524 7795 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:25.849468 7795 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:25.854437 7795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98be9a8be5f44d06a38564be07aab827: Generated new TSK 0
I20260619 18:46:25.855119 7795 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.143.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab.
Entry for principal kudu/127.21.143.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab.
I20260619 18:46:25.880265 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.65:0
--local_ip_for_outbound_sockets=127.21.143.65
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:34717
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:36105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:25.989128 7810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:25.989380 7810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:25.989423 7810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:25.993480 7810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:25.993551 7810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:25.993609 7810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.65
I20260619 18:46:25.999006 7810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:36105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:34717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.65
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:26.000098 7810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:26.001096 7810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:26.010579 7815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:26.010579 7818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:26.010685 7810 server_base.cc:1034] running on GCE node
W20260619 18:46:26.010607 7816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:26.011174 7810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:26.011859 7810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:26.013047 7810 hybrid_clock.cc:648] HybridClock initialized: now 1781894786013033 us; error 25 us; skew 500 ppm
Jun 19 18:46:26 dist-test-slave-mld5 krb5kdc[7703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894786, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:26.015700 7810 init.cc:377] Logged in from keytab as kudu/127.21.143.65@KRBTEST.COM (short username kudu)
I20260619 18:46:26.016515 7810 webserver.cc:460] Webserver started at http://127.21.143.65:41653/ using document root <none> and password file <none>
I20260619 18:46:26.017066 7810 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:26.017155 7810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:26.017377 7810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:26.019384 7810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-0/data/instance:
uuid: "e0df1325e4e241a1b860737f70abba10"
format_stamp: "Formatted at 2026-06-19 18:46:26 on dist-test-slave-mld5"
server_key: "0a056aecda54abc4aaa1508302f20561"
server_key_iv: "25396e7baa06baff379243c8b85454cb"
server_key_version: "encryptionkey@0"
I20260619 18:46:26.019958 7810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance:
uuid: "e0df1325e4e241a1b860737f70abba10"
format_stamp: "Formatted at 2026-06-19 18:46:26 on dist-test-slave-mld5"
server_key: "0a056aecda54abc4aaa1508302f20561"
server_key_iv: "25396e7baa06baff379243c8b85454cb"
server_key_version: "encryptionkey@0"
I20260619 18:46:26.023670 7810 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260619 18:46:26.025969 7825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:26.027138 7810 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:46:26.027276 7810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-0/wal
uuid: "e0df1325e4e241a1b860737f70abba10"
format_stamp: "Formatted at 2026-06-19 18:46:26 on dist-test-slave-mld5"
server_key: "0a056aecda54abc4aaa1508302f20561"
server_key_iv: "25396e7baa06baff379243c8b85454cb"
server_key_version: "encryptionkey@0"
I20260619 18:46:26.027441 7810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:26.046947 7810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:26.049887 7810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:26.050100 7810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:26.050683 7810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:46:26.051571 7810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:46:26.051641 7810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:26.051714 7810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:46:26.051751 7810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:26.060278 7810 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.65:37927
I20260619 18:46:26.060305 7938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.65:37927 every 8 connection(s)
I20260619 18:46:26.061136 7810 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
I20260619 18:46:26.066233 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 7810
I20260619 18:46:26.066390 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance
I20260619 18:46:26.066682 22077 external_mini_cluster.cc:1428] Setting key 202f40c6f07e81ee808b7aa928d82f4b
Jun 19 18:46:26 dist-test-slave-mld5 krb5kdc[7703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894786, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:46:26.075719 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:26.062868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:53403 (local address 127.21.143.126:34717)
0619 18:46:26.062999 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:26.063003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:26.063857 (+ 854us) server_negotiation.cc:407] Connection header received
0619 18:46:26.065172 (+ 1315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:26.065176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:26.065235 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:26.065349 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:26.066588 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.067341 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.068716 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.068952 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.069066 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:26.071578 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:26.071585 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:26.071591 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:26.071625 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:26.073532 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:26.074030 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:26.074033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:26.074035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:26.074136 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:26.074504 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:26.074507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:26.074509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:26.074758 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:26.074870 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:26.075375 (+ 505us) server_negotiation.cc:299] Negotiation successful
0619 18:46:26.075503 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
WARNING: no policy specified for kudu/127.21.143.66@KRBTEST.COM; defaulting to no policy
I20260619 18:46:26.076449 7941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:26.063240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:34717 (local address 127.21.143.65:53403)
0619 18:46:26.063718 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:26.063761 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:26.064966 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:26.065521 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:26.065528 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:26.065915 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:26.066431 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:26.066461 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.067465 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.067469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:26.068578 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:26.068588 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.069109 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.069114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:26.069280 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:26.069871 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:26.069888 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:26.071419 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:26.073668 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:26.073673 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:26.073683 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:26.073933 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:26.074239 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:26.074241 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:26.074243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:26.074404 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:26.074863 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:26.074867 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:26.075098 (+ 231us) client_negotiation.cc:769] Sending connection context
0619 18:46:26.075256 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 18:46:26.075553 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":108,"threads_started":1}
I20260619 18:46:26.077585 7939 heartbeater.cc:347] Connected to a master server at 127.21.143.126:34717
I20260619 18:46:26.077818 7939 heartbeater.cc:464] Registering TS with master...
I20260619 18:46:26.078269 7939 heartbeater.cc:511] Master 127.21.143.126:34717 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.66@KRBTEST.COM" created.
I20260619 18:46:26.079861 7748 ts_manager.cc:194] Registered new tserver with Master: e0df1325e4e241a1b860737f70abba10 (127.21.143.65:37927)
I20260619 18:46:26.081066 7748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.65@KRBTEST.COM'} at 127.21.143.65:53403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab.
Entry for principal kudu/127.21.143.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab.
I20260619 18:46:26.092159 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.66:0
--local_ip_for_outbound_sockets=127.21.143.66
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:34717
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:36105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:26.201524 7944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:26.201845 7944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:26.201923 7944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:26.206126 7944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:26.206210 7944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:26.206295 7944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.66
I20260619 18:46:26.211920 7944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:36105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:34717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.66
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:26.213039 7944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:26.214100 7944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:26.223973 7952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:26.224067 7950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:26.223999 7949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:26.224248 7944 server_base.cc:1034] running on GCE node
I20260619 18:46:26.224697 7944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:26.225353 7944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:26.226529 7944 hybrid_clock.cc:648] HybridClock initialized: now 1781894786226512 us; error 27 us; skew 500 ppm
Jun 19 18:46:26 dist-test-slave-mld5 krb5kdc[7703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894786, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:26.229275 7944 init.cc:377] Logged in from keytab as kudu/127.21.143.66@KRBTEST.COM (short username kudu)
I20260619 18:46:26.230108 7944 webserver.cc:460] Webserver started at http://127.21.143.66:34951/ using document root <none> and password file <none>
I20260619 18:46:26.230667 7944 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:26.230726 7944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:26.230901 7944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:26.233016 7944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-1/data/instance:
uuid: "5b23fc274d8a4bb4817f41b58d7ff0a5"
format_stamp: "Formatted at 2026-06-19 18:46:26 on dist-test-slave-mld5"
server_key: "337d2b12aa834e873714c0bced23dc72"
server_key_iv: "da75c98afcae35507d88d32455686a11"
server_key_version: "encryptionkey@0"
I20260619 18:46:26.233521 7944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance:
uuid: "5b23fc274d8a4bb4817f41b58d7ff0a5"
format_stamp: "Formatted at 2026-06-19 18:46:26 on dist-test-slave-mld5"
server_key: "337d2b12aa834e873714c0bced23dc72"
server_key_iv: "da75c98afcae35507d88d32455686a11"
server_key_version: "encryptionkey@0"
I20260619 18:46:26.237341 7944 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260619 18:46:26.239929 7959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:26.241637 7944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:46:26.241752 7944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-1/wal
uuid: "5b23fc274d8a4bb4817f41b58d7ff0a5"
format_stamp: "Formatted at 2026-06-19 18:46:26 on dist-test-slave-mld5"
server_key: "337d2b12aa834e873714c0bced23dc72"
server_key_iv: "da75c98afcae35507d88d32455686a11"
server_key_version: "encryptionkey@0"
I20260619 18:46:26.241856 7944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:26.260208 7944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:26.263520 7944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:26.263674 7944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:26.264209 7944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:46:26.264973 7944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:46:26.265027 7944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:26.265067 7944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:46:26.265091 7944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:26.273429 7944 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.66:42761
I20260619 18:46:26.273519 8072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.66:42761 every 8 connection(s)
I20260619 18:46:26.274327 7944 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
I20260619 18:46:26.278254 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 7944
I20260619 18:46:26.278398 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance
I20260619 18:46:26.278748 22077 external_mini_cluster.cc:1428] Setting key 1957013880a964ad1d3eea96c709f658
Jun 19 18:46:26 dist-test-slave-mld5 krb5kdc[7703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894786, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.143.67@KRBTEST.COM; defaulting to no policy
I20260619 18:46:26.289409 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:26.276135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:43957 (local address 127.21.143.126:34717)
0619 18:46:26.276296 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:26.276300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:26.277104 (+ 804us) server_negotiation.cc:407] Connection header received
0619 18:46:26.278413 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:26.278418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:26.278479 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:26.278582 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:26.279870 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.280548 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.282295 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.282697 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.282822 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:26.285713 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:26.285716 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:26.285718 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:26.285742 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:26.287182 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:26.287838 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:26.287841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:26.287842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:26.287915 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:26.288314 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:26.288316 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:26.288317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:26.288499 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:26.288599 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:26.289115 (+ 516us) server_negotiation.cc:299] Negotiation successful
0619 18:46:26.289238 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 18:46:26.290340 8075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:26.276519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:34717 (local address 127.21.143.66:43957)
0619 18:46:26.276953 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:26.276995 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:26.278178 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:26.278737 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:26.278744 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:26.279123 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:26.279735 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:26.279750 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.280690 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.280696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:26.282171 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:26.282181 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.282848 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.282851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:26.282993 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:26.283674 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:26.283691 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:26.285539 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:26.287345 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:26.287354 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:26.287376 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:26.287741 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:26.288020 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:26.288026 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:26.288031 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:26.288219 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:26.288609 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:26.288617 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:26.288863 (+ 246us) client_negotiation.cc:769] Sending connection context
0619 18:46:26.289051 (+ 188us) client_negotiation.cc:240] Negotiation successful
0619 18:46:26.289380 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":103,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.67@KRBTEST.COM" created.
I20260619 18:46:26.291626 8073 heartbeater.cc:347] Connected to a master server at 127.21.143.126:34717
I20260619 18:46:26.291872 8073 heartbeater.cc:464] Registering TS with master...
I20260619 18:46:26.292376 8073 heartbeater.cc:511] Master 127.21.143.126:34717 requested a full tablet report, sending...
I20260619 18:46:26.293541 7748 ts_manager.cc:194] Registered new tserver with Master: 5b23fc274d8a4bb4817f41b58d7ff0a5 (127.21.143.66:42761)
I20260619 18:46:26.294106 7748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.66@KRBTEST.COM'} at 127.21.143.66:43957
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab.
Entry for principal kudu/127.21.143.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab.
I20260619 18:46:26.306403 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.67:0
--local_ip_for_outbound_sockets=127.21.143.67
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:34717
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:36105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:26.415242 8078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:26.415573 8078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:26.415638 8078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:26.419750 8078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:26.419803 8078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:26.419911 8078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.67
I20260619 18:46:26.425407 8078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:36105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:34717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.67
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:26.426504 8078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:26.427623 8078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:26.437300 8084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:26.437294 8083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:26.437420 8078 server_base.cc:1034] running on GCE node
W20260619 18:46:26.437289 8086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:26.437974 8078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:26.438571 8078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:26.439726 8078 hybrid_clock.cc:648] HybridClock initialized: now 1781894786439713 us; error 24 us; skew 500 ppm
Jun 19 18:46:26 dist-test-slave-mld5 krb5kdc[7703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894786, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:26.442406 8078 init.cc:377] Logged in from keytab as kudu/127.21.143.67@KRBTEST.COM (short username kudu)
I20260619 18:46:26.443279 8078 webserver.cc:460] Webserver started at http://127.21.143.67:41999/ using document root <none> and password file <none>
I20260619 18:46:26.443889 8078 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:26.443950 8078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:26.444195 8078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:26.446302 8078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-2/data/instance:
uuid: "68b98cad161a4e5ba3b0aede21ceb1a7"
format_stamp: "Formatted at 2026-06-19 18:46:26 on dist-test-slave-mld5"
server_key: "018c0dfc86d6cc315008e4dba943e721"
server_key_iv: "7fed46e02fd0c0a0456d359df2bb4f24"
server_key_version: "encryptionkey@0"
I20260619 18:46:26.446866 8078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance:
uuid: "68b98cad161a4e5ba3b0aede21ceb1a7"
format_stamp: "Formatted at 2026-06-19 18:46:26 on dist-test-slave-mld5"
server_key: "018c0dfc86d6cc315008e4dba943e721"
server_key_iv: "7fed46e02fd0c0a0456d359df2bb4f24"
server_key_version: "encryptionkey@0"
I20260619 18:46:26.450587 8078 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260619 18:46:26.453101 8093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:26.454296 8078 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 18:46:26.454439 8078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-2/wal
uuid: "68b98cad161a4e5ba3b0aede21ceb1a7"
format_stamp: "Formatted at 2026-06-19 18:46:26 on dist-test-slave-mld5"
server_key: "018c0dfc86d6cc315008e4dba943e721"
server_key_iv: "7fed46e02fd0c0a0456d359df2bb4f24"
server_key_version: "encryptionkey@0"
I20260619 18:46:26.454555 8078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:26.472393 8078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:26.475279 8078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:26.475533 8078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:26.476106 8078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:46:26.476879 8078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:46:26.476948 8078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:26.477012 8078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:46:26.477067 8078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:26.485993 8078 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.67:38397
I20260619 18:46:26.486021 8206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.67:38397 every 8 connection(s)
I20260619 18:46:26.486835 8078 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
I20260619 18:46:26.492139 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 8078
I20260619 18:46:26.492295 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance
I20260619 18:46:26.492579 22077 external_mini_cluster.cc:1428] Setting key 2ba627d6acfce61b7a22cef18369cd0b
Jun 19 18:46:26 dist-test-slave-mld5 krb5kdc[7703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894786, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:46:26.500723 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:26.488493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:44193 (local address 127.21.143.126:34717)
0619 18:46:26.488630 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:26.488633 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:26.489395 (+ 762us) server_negotiation.cc:407] Connection header received
0619 18:46:26.490744 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:26.490748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:26.490802 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:26.490902 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:26.492138 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.492822 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.494246 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.494485 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.494598 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:26.497157 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:26.497161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:26.497163 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:26.497195 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:26.498550 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:26.499086 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:26.499090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:26.499092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:26.499175 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:26.499578 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:26.499582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:26.499583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:26.499773 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:26.499887 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:26.500427 (+ 540us) server_negotiation.cc:299] Negotiation successful
0619 18:46:26.500555 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:46:26.501614 8209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:26.488839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:34717 (local address 127.21.143.67:44193)
0619 18:46:26.489255 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:26.489294 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:26.490526 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:26.491076 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:26.491085 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:26.491512 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:26.491989 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:26.492002 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.492950 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.492954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:26.494129 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:26.494139 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.494642 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.494646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:26.494815 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:26.495418 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:26.495435 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:26.496980 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:26.498683 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:26.498689 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:26.498705 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:26.498973 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:26.499272 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:26.499276 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:26.499278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:26.499474 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:26.499886 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:26.499895 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:26.500171 (+ 276us) client_negotiation.cc:769] Sending connection context
0619 18:46:26.500374 (+ 203us) client_negotiation.cc:240] Negotiation successful
0619 18:46:26.500729 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":106,"threads_started":1}
I20260619 18:46:26.502964 8207 heartbeater.cc:347] Connected to a master server at 127.21.143.126:34717
I20260619 18:46:26.503255 8207 heartbeater.cc:464] Registering TS with master...
I20260619 18:46:26.503830 8207 heartbeater.cc:511] Master 127.21.143.126:34717 requested a full tablet report, sending...
I20260619 18:46:26.505533 7748 ts_manager.cc:194] Registered new tserver with Master: 68b98cad161a4e5ba3b0aede21ceb1a7 (127.21.143.67:38397)
I20260619 18:46:26.506435 7748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.67@KRBTEST.COM'} at 127.21.143.67:44193
I20260619 18:46:26.507400 22077 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:46:26.517477 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:26.509822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46554 (local address 127.21.143.126:34717)
0619 18:46:26.509972 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:26.509977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:26.510172 (+ 195us) server_negotiation.cc:407] Connection header received
0619 18:46:26.510245 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:26.510247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:26.510291 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:26.510366 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:26.511020 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.511720 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.512520 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.512728 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.512809 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:26.514244 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:26.514247 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:26.514249 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:26.514271 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:26.515863 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:26.516365 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:26.516367 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:26.516369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:26.516440 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:26.516704 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:26.516707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:26.516708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:26.516892 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:26.517004 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:26.517196 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 18:46:26.517269 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 18:46:26.520310 7748 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:46:26.523587 7748 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:46:26.537487 8221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:26.532002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59566 (local address 127.21.143.66:42761)
0619 18:46:26.532956 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:26.532969 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:26.532992 (+ 23us) server_negotiation.cc:407] Connection header received
0619 18:46:26.533068 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:26.533097 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:26.533271 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:26.533441 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:46:26.533675 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.534520 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.536058 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.536824 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.536953 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:26.537130 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:26.537262 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 18:46:26.537372 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":831,"thread_start_us":104,"threads_started":1}
I20260619 18:46:26.537489 8222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:26.531608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:42761 (local address 127.0.0.1:59566)
0619 18:46:26.532447 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:26.532460 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:26.532548 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:26.533428 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:26.533431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:26.533449 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:46:26.533559 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:26.533567 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.534711 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.534715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:26.535928 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:26.535938 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.536951 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.536955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:26.537061 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:26.537102 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 18:46:26.537224 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 18:46:26.537345 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"thread_start_us":90,"threads_started":1}
I20260619 18:46:26.537878 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:26.532215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:38397 (local address 127.0.0.1:60324)
0619 18:46:26.533440 (+ 1225us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:26.533452 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:26.533526 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:26.534001 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:26.534004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:26.534022 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:46:26.534101 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:26.534106 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.535475 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.535478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:26.536411 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:26.536418 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.537481 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.537484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:26.537576 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:26.537604 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 18:46:26.537677 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 18:46:26.537751 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1148,"thread_start_us":91,"threads_started":1}
I20260619 18:46:26.538228 8218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:26.531521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:37927 (local address 127.0.0.1:46048)
0619 18:46:26.532118 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:26.532165 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:26.532288 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:26.533092 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:26.533098 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:26.533122 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:46:26.533237 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:26.533241 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.534611 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.534616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:26.535787 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:26.535798 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.537849 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.537852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:26.537958 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:26.537983 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 18:46:26.538050 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 18:46:26.538116 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":80,"threads_started":1}
I20260619 18:46:26.538329 8223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:26.532460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60324 (local address 127.21.143.67:38397)
0619 18:46:26.532858 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:26.532869 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:26.533617 (+ 748us) server_negotiation.cc:407] Connection header received
0619 18:46:26.533693 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:26.533698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:26.533863 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:26.533998 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:46:26.534210 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.535336 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.536524 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.537210 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.537300 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:26.537423 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:26.538095 (+ 672us) server_negotiation.cc:299] Negotiation successful
0619 18:46:26.538188 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":167,"threads_started":1}
I20260619 18:46:26.538640 8219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:26.531619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46048 (local address 127.21.143.65:37927)
0619 18:46:26.532229 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:26.532245 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:26.532269 (+ 24us) server_negotiation.cc:407] Connection header received
0619 18:46:26.532502 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:26.532508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:26.532728 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:26.532877 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:46:26.533352 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.534478 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.535927 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.536824 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.537168 (+ 344us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:26.537311 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:26.538391 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0619 18:46:26.538495 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":60,"threads_started":1}
I20260619 18:46:26.540711 8141 tablet_service.cc:1467] Processing CreateTablet for tablet 351fd2b80e0446018f6805a9da8316ba (DEFAULT_TABLE table=test-table [id=6f740576dd1a48c4a8d9a1032bf9d942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:46:26.540736 8007 tablet_service.cc:1467] Processing CreateTablet for tablet 351fd2b80e0446018f6805a9da8316ba (DEFAULT_TABLE table=test-table [id=6f740576dd1a48c4a8d9a1032bf9d942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:46:26.541179 7873 tablet_service.cc:1467] Processing CreateTablet for tablet 351fd2b80e0446018f6805a9da8316ba (DEFAULT_TABLE table=test-table [id=6f740576dd1a48c4a8d9a1032bf9d942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:46:26.541563 8141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 351fd2b80e0446018f6805a9da8316ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:26.541746 8007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 351fd2b80e0446018f6805a9da8316ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:26.542009 7873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 351fd2b80e0446018f6805a9da8316ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:26.548147 8224 tablet_bootstrap.cc:492] T 351fd2b80e0446018f6805a9da8316ba P 68b98cad161a4e5ba3b0aede21ceb1a7: Bootstrap starting.
I20260619 18:46:26.548936 8226 tablet_bootstrap.cc:492] T 351fd2b80e0446018f6805a9da8316ba P 5b23fc274d8a4bb4817f41b58d7ff0a5: Bootstrap starting.
I20260619 18:46:26.549358 8225 tablet_bootstrap.cc:492] T 351fd2b80e0446018f6805a9da8316ba P e0df1325e4e241a1b860737f70abba10: Bootstrap starting.
I20260619 18:46:26.550380 8224 tablet_bootstrap.cc:654] T 351fd2b80e0446018f6805a9da8316ba P 68b98cad161a4e5ba3b0aede21ceb1a7: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:26.550992 8226 tablet_bootstrap.cc:654] T 351fd2b80e0446018f6805a9da8316ba P 5b23fc274d8a4bb4817f41b58d7ff0a5: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:26.551014 8225 tablet_bootstrap.cc:654] T 351fd2b80e0446018f6805a9da8316ba P e0df1325e4e241a1b860737f70abba10: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:26.551854 8224 log.cc:824] T 351fd2b80e0446018f6805a9da8316ba P 68b98cad161a4e5ba3b0aede21ceb1a7: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:26.552307 8226 log.cc:824] T 351fd2b80e0446018f6805a9da8316ba P 5b23fc274d8a4bb4817f41b58d7ff0a5: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:26.552322 8225 log.cc:824] T 351fd2b80e0446018f6805a9da8316ba P e0df1325e4e241a1b860737f70abba10: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:26.553962 8225 tablet_bootstrap.cc:492] T 351fd2b80e0446018f6805a9da8316ba P e0df1325e4e241a1b860737f70abba10: No bootstrap required, opened a new log
I20260619 18:46:26.554160 8225 ts_tablet_manager.cc:1397] T 351fd2b80e0446018f6805a9da8316ba P e0df1325e4e241a1b860737f70abba10: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 18:46:26.554229 8226 tablet_bootstrap.cc:492] T 351fd2b80e0446018f6805a9da8316ba P 5b23fc274d8a4bb4817f41b58d7ff0a5: No bootstrap required, opened a new log
I20260619 18:46:26.554394 8226 ts_tablet_manager.cc:1397] T 351fd2b80e0446018f6805a9da8316ba P 5b23fc274d8a4bb4817f41b58d7ff0a5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 18:46:26.555182 8224 tablet_bootstrap.cc:492] T 351fd2b80e0446018f6805a9da8316ba P 68b98cad161a4e5ba3b0aede21ceb1a7: No bootstrap required, opened a new log
I20260619 18:46:26.555369 8224 ts_tablet_manager.cc:1397] T 351fd2b80e0446018f6805a9da8316ba P 68b98cad161a4e5ba3b0aede21ceb1a7: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 18:46:26.557096 8225 raft_consensus.cc:348] T 351fd2b80e0446018f6805a9da8316ba P e0df1325e4e241a1b860737f70abba10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0df1325e4e241a1b860737f70abba10" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 37927 } } peers { permanent_uuid: "5b23fc274d8a4bb4817f41b58d7ff0a5" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42761 } } peers { permanent_uuid: "68b98cad161a4e5ba3b0aede21ceb1a7" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38397 } }
I20260619 18:46:26.557366 8225 raft_consensus.cc:374] T 351fd2b80e0446018f6805a9da8316ba P e0df1325e4e241a1b860737f70abba10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:26.557430 8225 raft_consensus.cc:729] T 351fd2b80e0446018f6805a9da8316ba P e0df1325e4e241a1b860737f70abba10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0df1325e4e241a1b860737f70abba10, State: Initialized, Role: FOLLOWER
I20260619 18:46:26.557868 8225 consensus_queue.cc:260] T 351fd2b80e0446018f6805a9da8316ba P e0df1325e4e241a1b860737f70abba10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0df1325e4e241a1b860737f70abba10" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 37927 } } peers { permanent_uuid: "5b23fc274d8a4bb4817f41b58d7ff0a5" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42761 } } peers { permanent_uuid: "68b98cad161a4e5ba3b0aede21ceb1a7" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38397 } }
I20260619 18:46:26.558143 8226 raft_consensus.cc:348] T 351fd2b80e0446018f6805a9da8316ba P 5b23fc274d8a4bb4817f41b58d7ff0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0df1325e4e241a1b860737f70abba10" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 37927 } } peers { permanent_uuid: "5b23fc274d8a4bb4817f41b58d7ff0a5" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42761 } } peers { permanent_uuid: "68b98cad161a4e5ba3b0aede21ceb1a7" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38397 } }
I20260619 18:46:26.558373 8226 raft_consensus.cc:374] T 351fd2b80e0446018f6805a9da8316ba P 5b23fc274d8a4bb4817f41b58d7ff0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:26.558415 8226 raft_consensus.cc:729] T 351fd2b80e0446018f6805a9da8316ba P 5b23fc274d8a4bb4817f41b58d7ff0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b23fc274d8a4bb4817f41b58d7ff0a5, State: Initialized, Role: FOLLOWER
I20260619 18:46:26.558815 8226 consensus_queue.cc:260] T 351fd2b80e0446018f6805a9da8316ba P 5b23fc274d8a4bb4817f41b58d7ff0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0df1325e4e241a1b860737f70abba10" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 37927 } } peers { permanent_uuid: "5b23fc274d8a4bb4817f41b58d7ff0a5" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42761 } } peers { permanent_uuid: "68b98cad161a4e5ba3b0aede21ceb1a7" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38397 } }
I20260619 18:46:26.558954 8225 ts_tablet_manager.cc:1428] T 351fd2b80e0446018f6805a9da8316ba P e0df1325e4e241a1b860737f70abba10: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 18:46:26.558944 8224 raft_consensus.cc:348] T 351fd2b80e0446018f6805a9da8316ba P 68b98cad161a4e5ba3b0aede21ceb1a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0df1325e4e241a1b860737f70abba10" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 37927 } } peers { permanent_uuid: "5b23fc274d8a4bb4817f41b58d7ff0a5" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42761 } } peers { permanent_uuid: "68b98cad161a4e5ba3b0aede21ceb1a7" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38397 } }
I20260619 18:46:26.559312 7939 heartbeater.cc:503] Master 127.21.143.126:34717 was elected leader, sending a full tablet report...
I20260619 18:46:26.559758 8226 ts_tablet_manager.cc:1428] T 351fd2b80e0446018f6805a9da8316ba P 5b23fc274d8a4bb4817f41b58d7ff0a5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 18:46:26.560086 8073 heartbeater.cc:503] Master 127.21.143.126:34717 was elected leader, sending a full tablet report...
I20260619 18:46:26.560211 8224 raft_consensus.cc:374] T 351fd2b80e0446018f6805a9da8316ba P 68b98cad161a4e5ba3b0aede21ceb1a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:26.560281 8224 raft_consensus.cc:729] T 351fd2b80e0446018f6805a9da8316ba P 68b98cad161a4e5ba3b0aede21ceb1a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68b98cad161a4e5ba3b0aede21ceb1a7, State: Initialized, Role: FOLLOWER
I20260619 18:46:26.560629 8224 consensus_queue.cc:260] T 351fd2b80e0446018f6805a9da8316ba P 68b98cad161a4e5ba3b0aede21ceb1a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0df1325e4e241a1b860737f70abba10" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 37927 } } peers { permanent_uuid: "5b23fc274d8a4bb4817f41b58d7ff0a5" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42761 } } peers { permanent_uuid: "68b98cad161a4e5ba3b0aede21ceb1a7" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38397 } }
I20260619 18:46:26.561151 8207 heartbeater.cc:503] Master 127.21.143.126:34717 was elected leader, sending a full tablet report...
I20260619 18:46:26.561406 8224 ts_tablet_manager.cc:1428] T 351fd2b80e0446018f6805a9da8316ba P 68b98cad161a4e5ba3b0aede21ceb1a7: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
W20260619 18:46:26.562230 7940 tablet.cc:2369] T 351fd2b80e0446018f6805a9da8316ba P e0df1325e4e241a1b860737f70abba10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:46:26.562420 8230 raft_consensus.cc:482] T 351fd2b80e0446018f6805a9da8316ba P e0df1325e4e241a1b860737f70abba10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:46:26.562557 8230 raft_consensus.cc:504] T 351fd2b80e0446018f6805a9da8316ba P e0df1325e4e241a1b860737f70abba10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0df1325e4e241a1b860737f70abba10" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 37927 } } peers { permanent_uuid: "5b23fc274d8a4bb4817f41b58d7ff0a5" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42761 } } peers { permanent_uuid: "68b98cad161a4e5ba3b0aede21ceb1a7" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38397 } }
I20260619 18:46:26.563699 8230 leader_election.cc:288] T 351fd2b80e0446018f6805a9da8316ba P e0df1325e4e241a1b860737f70abba10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b23fc274d8a4bb4817f41b58d7ff0a5 (127.21.143.66:42761), 68b98cad161a4e5ba3b0aede21ceb1a7 (127.21.143.67:38397)
I20260619 18:46:26.569779 7941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:26.564594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:38397 (local address 127.21.143.65:39971)
0619 18:46:26.566385 (+ 1791us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:26.566401 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:26.566482 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:26.566862 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:26.566868 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:26.566887 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:46:26.567000 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:26.567007 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.567992 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.567996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:26.568747 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:26.568756 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.569369 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.569372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:26.569470 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:26.569491 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 18:46:26.569579 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 18:46:26.569650 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":112,"threads_started":1}
I20260619 18:46:26.569774 8223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:26.566131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:39971 (local address 127.21.143.67:38397)
0619 18:46:26.566334 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:26.566339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:26.566579 (+ 240us) server_negotiation.cc:407] Connection header received
0619 18:46:26.566623 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:26.566625 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:26.566669 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:26.566740 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:46:26.567125 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.567888 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.568863 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.569247 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.569368 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:26.569418 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:26.569606 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 18:46:26.569665 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260619 18:46:26.570281 8221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:26.564527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:34561 (local address 127.21.143.66:42761)
0619 18:46:26.564643 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:26.564646 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:26.567284 (+ 2638us) server_negotiation.cc:407] Connection header received
0619 18:46:26.567393 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:26.567396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:26.567440 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:26.567521 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:46:26.567871 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.568343 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.569354 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.569759 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.569880 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:26.569923 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:26.570116 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 18:46:26.570190 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 18:46:26.570372 8161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "351fd2b80e0446018f6805a9da8316ba" candidate_uuid: "e0df1325e4e241a1b860737f70abba10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68b98cad161a4e5ba3b0aede21ceb1a7" is_pre_election: true
I20260619 18:46:26.570616 8161 raft_consensus.cc:2454] T 351fd2b80e0446018f6805a9da8316ba P 68b98cad161a4e5ba3b0aede21ceb1a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0df1325e4e241a1b860737f70abba10 in term 0.
I20260619 18:46:26.570640 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:26.564370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:42761 (local address 127.21.143.65:34561)
0619 18:46:26.567194 (+ 2824us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:26.567206 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:26.567285 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:26.567638 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:26.567641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:26.567659 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:46:26.567764 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:26.567768 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.568458 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.568461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:26.569224 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:26.569232 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.569872 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.569876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:26.569974 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:26.569999 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 18:46:26.570419 (+ 420us) client_negotiation.cc:240] Negotiation successful
0619 18:46:26.570518 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473}
I20260619 18:46:26.571125 7827 leader_election.cc:302] T 351fd2b80e0446018f6805a9da8316ba P e0df1325e4e241a1b860737f70abba10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68b98cad161a4e5ba3b0aede21ceb1a7, e0df1325e4e241a1b860737f70abba10; no voters:
I20260619 18:46:26.571358 8027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "351fd2b80e0446018f6805a9da8316ba" candidate_uuid: "e0df1325e4e241a1b860737f70abba10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b23fc274d8a4bb4817f41b58d7ff0a5" is_pre_election: true
I20260619 18:46:26.571403 8230 raft_consensus.cc:2781] T 351fd2b80e0446018f6805a9da8316ba P e0df1325e4e241a1b860737f70abba10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:46:26.571485 8230 raft_consensus.cc:482] T 351fd2b80e0446018f6805a9da8316ba P e0df1325e4e241a1b860737f70abba10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:46:26.571543 8230 raft_consensus.cc:3037] T 351fd2b80e0446018f6805a9da8316ba P e0df1325e4e241a1b860737f70abba10 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:26.571622 8027 raft_consensus.cc:2454] T 351fd2b80e0446018f6805a9da8316ba P 5b23fc274d8a4bb4817f41b58d7ff0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0df1325e4e241a1b860737f70abba10 in term 0.
I20260619 18:46:26.572609 8230 raft_consensus.cc:504] T 351fd2b80e0446018f6805a9da8316ba P e0df1325e4e241a1b860737f70abba10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0df1325e4e241a1b860737f70abba10" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 37927 } } peers { permanent_uuid: "5b23fc274d8a4bb4817f41b58d7ff0a5" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42761 } } peers { permanent_uuid: "68b98cad161a4e5ba3b0aede21ceb1a7" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38397 } }
I20260619 18:46:26.572922 8230 leader_election.cc:288] T 351fd2b80e0446018f6805a9da8316ba P e0df1325e4e241a1b860737f70abba10 [CANDIDATE]: Term 1 election: Requested vote from peers 5b23fc274d8a4bb4817f41b58d7ff0a5 (127.21.143.66:42761), 68b98cad161a4e5ba3b0aede21ceb1a7 (127.21.143.67:38397)
I20260619 18:46:26.573246 8027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "351fd2b80e0446018f6805a9da8316ba" candidate_uuid: "e0df1325e4e241a1b860737f70abba10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b23fc274d8a4bb4817f41b58d7ff0a5"
I20260619 18:46:26.573338 8161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "351fd2b80e0446018f6805a9da8316ba" candidate_uuid: "e0df1325e4e241a1b860737f70abba10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68b98cad161a4e5ba3b0aede21ceb1a7"
I20260619 18:46:26.573362 8027 raft_consensus.cc:3037] T 351fd2b80e0446018f6805a9da8316ba P 5b23fc274d8a4bb4817f41b58d7ff0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:26.573459 8161 raft_consensus.cc:3037] T 351fd2b80e0446018f6805a9da8316ba P 68b98cad161a4e5ba3b0aede21ceb1a7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:26.574618 8027 raft_consensus.cc:2454] T 351fd2b80e0446018f6805a9da8316ba P 5b23fc274d8a4bb4817f41b58d7ff0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0df1325e4e241a1b860737f70abba10 in term 1.
I20260619 18:46:26.574689 8161 raft_consensus.cc:2454] T 351fd2b80e0446018f6805a9da8316ba P 68b98cad161a4e5ba3b0aede21ceb1a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0df1325e4e241a1b860737f70abba10 in term 1.
I20260619 18:46:26.575035 7827 leader_election.cc:302] T 351fd2b80e0446018f6805a9da8316ba P e0df1325e4e241a1b860737f70abba10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68b98cad161a4e5ba3b0aede21ceb1a7, e0df1325e4e241a1b860737f70abba10; no voters:
I20260619 18:46:26.575212 8230 raft_consensus.cc:2781] T 351fd2b80e0446018f6805a9da8316ba P e0df1325e4e241a1b860737f70abba10 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:26.575443 8230 raft_consensus.cc:686] T 351fd2b80e0446018f6805a9da8316ba P e0df1325e4e241a1b860737f70abba10 [term 1 LEADER]: Becoming Leader. State: Replica: e0df1325e4e241a1b860737f70abba10, State: Running, Role: LEADER
I20260619 18:46:26.575728 8230 consensus_queue.cc:237] T 351fd2b80e0446018f6805a9da8316ba P e0df1325e4e241a1b860737f70abba10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0df1325e4e241a1b860737f70abba10" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 37927 } } peers { permanent_uuid: "5b23fc274d8a4bb4817f41b58d7ff0a5" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42761 } } peers { permanent_uuid: "68b98cad161a4e5ba3b0aede21ceb1a7" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38397 } }
I20260619 18:46:26.578539 7746 catalog_manager.cc:5495] T 351fd2b80e0446018f6805a9da8316ba P e0df1325e4e241a1b860737f70abba10 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0df1325e4e241a1b860737f70abba10 (127.21.143.65). New cstate: current_term: 1 leader_uuid: "e0df1325e4e241a1b860737f70abba10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0df1325e4e241a1b860737f70abba10" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 37927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b23fc274d8a4bb4817f41b58d7ff0a5" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68b98cad161a4e5ba3b0aede21ceb1a7" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38397 } health_report { overall_health: UNKNOWN } } }
I20260619 18:46:26.591915 8219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:26.588632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46062 (local address 127.21.143.65:37927)
0619 18:46:26.588753 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:26.588756 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:26.588784 (+ 28us) server_negotiation.cc:407] Connection header received
0619 18:46:26.588908 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:26.588911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:26.588959 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:26.589054 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:46:26.589253 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.589718 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.590474 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.590681 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.590769 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:26.591046 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:46:26.591496 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:46:26.591605 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:26.591693 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 18:46:26.591747 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 18:46:26.597196 8161 raft_consensus.cc:1261] T 351fd2b80e0446018f6805a9da8316ba P 68b98cad161a4e5ba3b0aede21ceb1a7 [term 1 FOLLOWER]: Refusing update from remote peer e0df1325e4e241a1b860737f70abba10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:46:26.597235 8027 raft_consensus.cc:1261] T 351fd2b80e0446018f6805a9da8316ba P 5b23fc274d8a4bb4817f41b58d7ff0a5 [term 1 FOLLOWER]: Refusing update from remote peer e0df1325e4e241a1b860737f70abba10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:46:26.597853 8230 consensus_queue.cc:1035] T 351fd2b80e0446018f6805a9da8316ba P e0df1325e4e241a1b860737f70abba10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b23fc274d8a4bb4817f41b58d7ff0a5" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:46:26.598027 8234 consensus_queue.cc:1035] T 351fd2b80e0446018f6805a9da8316ba P e0df1325e4e241a1b860737f70abba10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68b98cad161a4e5ba3b0aede21ceb1a7" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:46:26.606287 8238 mvcc.cc:205] Tried to move back new op lower bound from 7298641045895847936 to 7298641045816684544. Current Snapshot: MvccSnapshot[applied={T|T < 7298641045895847936 or (T in {7298641045895847936})}]
I20260619 18:46:26.613063 7746 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:46:26.613219 7746 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:46:26.615470 7746 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98be9a8be5f44d06a38564be07aab827: Sending DeleteTablet for 3 replicas of tablet 351fd2b80e0446018f6805a9da8316ba
I20260619 18:46:26.616151 7873 tablet_service.cc:1514] Processing DeleteTablet for tablet 351fd2b80e0446018f6805a9da8316ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:46:26 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:46048
I20260619 18:46:26.616242 8141 tablet_service.cc:1514] Processing DeleteTablet for tablet 351fd2b80e0446018f6805a9da8316ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:46:26 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:60324
I20260619 18:46:26.616341 8007 tablet_service.cc:1514] Processing DeleteTablet for tablet 351fd2b80e0446018f6805a9da8316ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:46:26 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:59566
I20260619 18:46:26.616748 8248 tablet_replica.cc:330] T 351fd2b80e0446018f6805a9da8316ba P 68b98cad161a4e5ba3b0aede21ceb1a7: stopping tablet replica
I20260619 18:46:26.616954 8249 tablet_replica.cc:330] T 351fd2b80e0446018f6805a9da8316ba P 5b23fc274d8a4bb4817f41b58d7ff0a5: stopping tablet replica
I20260619 18:46:26.617179 8248 raft_consensus.cc:2229] T 351fd2b80e0446018f6805a9da8316ba P 68b98cad161a4e5ba3b0aede21ceb1a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:46:26.617174 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 7810
I20260619 18:46:26.617204 8249 raft_consensus.cc:2229] T 351fd2b80e0446018f6805a9da8316ba P 5b23fc274d8a4bb4817f41b58d7ff0a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:46:26.618214 8248 raft_consensus.cc:2258] T 351fd2b80e0446018f6805a9da8316ba P 68b98cad161a4e5ba3b0aede21ceb1a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:46:26.618214 8249 raft_consensus.cc:2258] T 351fd2b80e0446018f6805a9da8316ba P 5b23fc274d8a4bb4817f41b58d7ff0a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:46:26.619629 8249 ts_tablet_manager.cc:1905] T 351fd2b80e0446018f6805a9da8316ba P 5b23fc274d8a4bb4817f41b58d7ff0a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:46:26.620589 8248 ts_tablet_manager.cc:1905] T 351fd2b80e0446018f6805a9da8316ba P 68b98cad161a4e5ba3b0aede21ceb1a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:46:26.622697 8248 ts_tablet_manager.cc:1918] T 351fd2b80e0446018f6805a9da8316ba P 68b98cad161a4e5ba3b0aede21ceb1a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:46:26.622809 8248 log.cc:1196] T 351fd2b80e0446018f6805a9da8316ba P 68b98cad161a4e5ba3b0aede21ceb1a7: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-2/wal/wals/351fd2b80e0446018f6805a9da8316ba
I20260619 18:46:26.623064 8249 ts_tablet_manager.cc:1918] T 351fd2b80e0446018f6805a9da8316ba P 5b23fc274d8a4bb4817f41b58d7ff0a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:46:26.623159 8249 log.cc:1196] T 351fd2b80e0446018f6805a9da8316ba P 5b23fc274d8a4bb4817f41b58d7ff0a5: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894542572984-22077-0/minicluster-data/ts-1/wal/wals/351fd2b80e0446018f6805a9da8316ba
I20260619 18:46:26.623209 8248 ts_tablet_manager.cc:1939] T 351fd2b80e0446018f6805a9da8316ba P 68b98cad161a4e5ba3b0aede21ceb1a7: Deleting consensus metadata
I20260619 18:46:26.623538 8249 ts_tablet_manager.cc:1939] T 351fd2b80e0446018f6805a9da8316ba P 5b23fc274d8a4bb4817f41b58d7ff0a5: Deleting consensus metadata
I20260619 18:46:26.624346 7733 catalog_manager.cc:4841] TS 68b98cad161a4e5ba3b0aede21ceb1a7 (127.21.143.67:38397): tablet 351fd2b80e0446018f6805a9da8316ba (table test-table [id=6f740576dd1a48c4a8d9a1032bf9d942]) successfully deleted
I20260619 18:46:26.624548 7733 catalog_manager.cc:4841] TS 5b23fc274d8a4bb4817f41b58d7ff0a5 (127.21.143.66:42761): tablet 351fd2b80e0446018f6805a9da8316ba (table test-table [id=6f740576dd1a48c4a8d9a1032bf9d942]) successfully deleted
W20260619 18:46:26.625593 7735 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.143.65:37927 (error 108)
I20260619 18:46:26.625734 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 7944
I20260619 18:46:26.627197 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:26.626877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:37927 (local address 127.0.0.1:46066)
0619 18:46:26.627028 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:26.627102 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.65:37927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 18:46:26.627358 7735 catalog_manager.cc:4584] TS e0df1325e4e241a1b860737f70abba10 (127.21.143.65:37927): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 351fd2b80e0446018f6805a9da8316ba: Network error: Client connection negotiation failed: client connection to 127.21.143.65:37927: connect: Connection refused (error 111)
I20260619 18:46:26.631645 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 8078
I20260619 18:46:26.637075 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 7716
2026-06-19T18:46:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2176 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T18:46:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:26Z Disabled control of system clock
I20260619 18:46:26.668342 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:42633
--webserver_interface=127.21.143.126
--webserver_port=0
--builtin_ntp_servers=127.21.143.84:46619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:42633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 18:46:26.776232 8254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:26.776513 8254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:26.776551 8254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:26.780488 8254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:26.780536 8254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:26.780552 8254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:26.780568 8254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:26.780583 8254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:26.785166 8254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:46619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:42633
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:42633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:26.786185 8254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:26.787580 8254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:26.796154 8262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:26.796113 8260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:26.796113 8259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:26.796445 8254 server_base.cc:1034] running on GCE node
I20260619 18:46:26.797034 8254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:26.798159 8254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:26.799494 8254 hybrid_clock.cc:648] HybridClock initialized: now 1781894786799459 us; error 37 us; skew 500 ppm
I20260619 18:46:26.801329 8254 webserver.cc:460] Webserver started at http://127.21.143.126:44565/ using document root <none> and password file <none>
I20260619 18:46:26.801942 8254 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:26.802008 8254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:26.802261 8254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:26.804471 8254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "3d37ae677ab14bbaaefb58212b5c5e49"
format_stamp: "Formatted at 2026-06-19 18:46:26 on dist-test-slave-mld5"
server_key: "6eff8826e2da9f414936fa89e4cacd10"
server_key_iv: "b26dee2f8a45c7a7edc15dd101b795af"
server_key_version: "encryptionkey@0"
I20260619 18:46:26.805061 8254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "3d37ae677ab14bbaaefb58212b5c5e49"
format_stamp: "Formatted at 2026-06-19 18:46:26 on dist-test-slave-mld5"
server_key: "6eff8826e2da9f414936fa89e4cacd10"
server_key_iv: "b26dee2f8a45c7a7edc15dd101b795af"
server_key_version: "encryptionkey@0"
I20260619 18:46:26.808979 8254 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:46:26.811574 8268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:26.812813 8254 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:46:26.812963 8254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "3d37ae677ab14bbaaefb58212b5c5e49"
format_stamp: "Formatted at 2026-06-19 18:46:26 on dist-test-slave-mld5"
server_key: "6eff8826e2da9f414936fa89e4cacd10"
server_key_iv: "b26dee2f8a45c7a7edc15dd101b795af"
server_key_version: "encryptionkey@0"
I20260619 18:46:26.813082 8254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:26.826587 8254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:26.827280 8254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:26.827555 8254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:26.834738 8254 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:42633
I20260619 18:46:26.834764 8320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:42633 every 8 connection(s)
I20260619 18:46:26.835741 8254 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:26.838797 8321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:26.843600 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 8254
I20260619 18:46:26.843776 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:46:26.844075 22077 external_mini_cluster.cc:1428] Setting key 44d5a20cc8f0b56b631cd0a3cee0e73a
I20260619 18:46:26.844308 8321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d37ae677ab14bbaaefb58212b5c5e49: Bootstrap starting.
I20260619 18:46:26.846669 8321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d37ae677ab14bbaaefb58212b5c5e49: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:26.847437 8321 log.cc:824] T 00000000000000000000000000000000 P 3d37ae677ab14bbaaefb58212b5c5e49: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:26.849166 8321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d37ae677ab14bbaaefb58212b5c5e49: No bootstrap required, opened a new log
I20260619 18:46:26.852810 8324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:26.845410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57810 (local address 127.21.143.126:42633)
0619 18:46:26.846016 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:26.846030 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:26.846076 (+ 46us) server_negotiation.cc:407] Connection header received
0619 18:46:26.846659 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:26.846674 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:26.847098 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:26.847433 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:26.847829 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.848749 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.849669 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.850120 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.850286 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:26.851118 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:26.851123 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:26.851135 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:26.851552 (+ 417us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:26.851604 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:26.851624 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:26.851723 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:26.851871 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 18:46:26.852122 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":122,"threads_started":1}
I20260619 18:46:26.852792 8321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d37ae677ab14bbaaefb58212b5c5e49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d37ae677ab14bbaaefb58212b5c5e49" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42633 } }
I20260619 18:46:26.853063 8321 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d37ae677ab14bbaaefb58212b5c5e49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:26.853132 8321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d37ae677ab14bbaaefb58212b5c5e49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d37ae677ab14bbaaefb58212b5c5e49, State: Initialized, Role: FOLLOWER
I20260619 18:46:26.853570 8321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d37ae677ab14bbaaefb58212b5c5e49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d37ae677ab14bbaaefb58212b5c5e49" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42633 } }
I20260619 18:46:26.853708 8321 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d37ae677ab14bbaaefb58212b5c5e49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:26.853763 8321 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d37ae677ab14bbaaefb58212b5c5e49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:26.853848 8321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d37ae677ab14bbaaefb58212b5c5e49 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:26.854730 8321 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d37ae677ab14bbaaefb58212b5c5e49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d37ae677ab14bbaaefb58212b5c5e49" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42633 } }
I20260619 18:46:26.855103 8321 leader_election.cc:302] T 00000000000000000000000000000000 P 3d37ae677ab14bbaaefb58212b5c5e49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d37ae677ab14bbaaefb58212b5c5e49; no voters:
I20260619 18:46:26.855445 8321 leader_election.cc:288] T 00000000000000000000000000000000 P 3d37ae677ab14bbaaefb58212b5c5e49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:26.855547 8326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d37ae677ab14bbaaefb58212b5c5e49 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:26.855777 8326 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d37ae677ab14bbaaefb58212b5c5e49 [term 1 LEADER]: Becoming Leader. State: Replica: 3d37ae677ab14bbaaefb58212b5c5e49, State: Running, Role: LEADER
I20260619 18:46:26.856093 8326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d37ae677ab14bbaaefb58212b5c5e49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d37ae677ab14bbaaefb58212b5c5e49" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42633 } }
I20260619 18:46:26.856621 8321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d37ae677ab14bbaaefb58212b5c5e49 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:26.857649 8327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d37ae677ab14bbaaefb58212b5c5e49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d37ae677ab14bbaaefb58212b5c5e49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d37ae677ab14bbaaefb58212b5c5e49" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42633 } } }
I20260619 18:46:26.857771 8327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d37ae677ab14bbaaefb58212b5c5e49 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:26.858047 8328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d37ae677ab14bbaaefb58212b5c5e49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d37ae677ab14bbaaefb58212b5c5e49. Latest consensus state: current_term: 1 leader_uuid: "3d37ae677ab14bbaaefb58212b5c5e49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d37ae677ab14bbaaefb58212b5c5e49" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 42633 } } }
I20260619 18:46:26.858141 8328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d37ae677ab14bbaaefb58212b5c5e49 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:26.858233 8335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:26.861303 8335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:26.866391 8335 catalog_manager.cc:1347] Generated new cluster ID: 30a385575854412bb029b8f8f6f816ec
I20260619 18:46:26.866451 8335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:26.887259 8335 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:26.888563 8335 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:26.895437 8335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d37ae677ab14bbaaefb58212b5c5e49: Generated new TSK 0
I20260619 18:46:26.896210 8335 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:26.902772 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.65:0
--local_ip_for_outbound_sockets=127.21.143.65
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:42633
--builtin_ntp_servers=127.21.143.84:46619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 18:46:27.011847 8345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:27.012126 8345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:27.012217 8345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:27.016273 8345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:27.016340 8345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:27.016436 8345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.65
I20260619 18:46:27.021193 8345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:46619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:42633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.65
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:27.022228 8345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:27.023689 8345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:27.032954 8350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:27.033092 8345 server_base.cc:1034] running on GCE node
W20260619 18:46:27.032966 8353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:27.032969 8351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:27.033660 8345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:27.034271 8345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:27.035456 8345 hybrid_clock.cc:648] HybridClock initialized: now 1781894787035432 us; error 30 us; skew 500 ppm
I20260619 18:46:27.037263 8345 webserver.cc:460] Webserver started at http://127.21.143.65:33751/ using document root <none> and password file <none>
I20260619 18:46:27.037838 8345 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:27.037930 8345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:27.038180 8345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:27.040246 8345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-0/data/instance:
uuid: "09fdbce739984769ad6edf4e85bf30ca"
format_stamp: "Formatted at 2026-06-19 18:46:27 on dist-test-slave-mld5"
server_key: "609f2ab23e052cde28fa5d68d27e1a64"
server_key_iv: "0d4d4361e547500717148ad3e7b0307e"
server_key_version: "encryptionkey@0"
I20260619 18:46:27.040794 8345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance:
uuid: "09fdbce739984769ad6edf4e85bf30ca"
format_stamp: "Formatted at 2026-06-19 18:46:27 on dist-test-slave-mld5"
server_key: "609f2ab23e052cde28fa5d68d27e1a64"
server_key_iv: "0d4d4361e547500717148ad3e7b0307e"
server_key_version: "encryptionkey@0"
I20260619 18:46:27.044330 8345 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:46:27.046561 8359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:27.047777 8345 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:46:27.047905 8345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-0/wal
uuid: "09fdbce739984769ad6edf4e85bf30ca"
format_stamp: "Formatted at 2026-06-19 18:46:27 on dist-test-slave-mld5"
server_key: "609f2ab23e052cde28fa5d68d27e1a64"
server_key_iv: "0d4d4361e547500717148ad3e7b0307e"
server_key_version: "encryptionkey@0"
I20260619 18:46:27.048009 8345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:27.061757 8345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:27.062417 8345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:27.062623 8345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:27.063179 8345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:46:27.064116 8345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:46:27.064188 8345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:27.064255 8345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:46:27.064303 8345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:27.072781 8345 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.65:45485
I20260619 18:46:27.072811 8472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.65:45485 every 8 connection(s)
I20260619 18:46:27.073683 8345 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
I20260619 18:46:27.078835 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 8345
I20260619 18:46:27.078969 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance
I20260619 18:46:27.079246 22077 external_mini_cluster.cc:1428] Setting key 4ab50098142f06f402d07742f854304e
I20260619 18:46:27.081184 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.66:0
--local_ip_for_outbound_sockets=127.21.143.66
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:42633
--builtin_ntp_servers=127.21.143.84:46619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 18:46:27.083555 8324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:27.075495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:50713 (local address 127.21.143.126:42633)
0619 18:46:27.075655 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:27.075659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:27.076720 (+ 1061us) server_negotiation.cc:407] Connection header received
0619 18:46:27.078219 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:27.078224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:27.078315 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:27.078421 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:27.079590 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:27.080214 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:27.081516 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:27.081765 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:27.081878 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:27.083016 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:27.083019 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:27.083022 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:27.083032 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:27.083053 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:27.083061 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:27.083120 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:27.083346 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 18:46:27.083405 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 18:46:27.084430 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:27.075914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42633 (local address 127.21.143.65:50713)
0619 18:46:27.076553 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:27.076591 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:27.077941 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:27.078618 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:27.078629 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:27.078761 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:27.079431 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:27.079452 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:27.080344 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:27.080347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:27.081391 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:27.081401 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:27.081915 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:27.081919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:27.082096 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:27.082718 (+ 622us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:46:27.082721 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:27.082776 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:46:27.082859 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:27.083145 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:27.083155 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:27.083163 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 18:46:27.083322 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 18:46:27.083634 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":110,"threads_started":1}
I20260619 18:46:27.085669 8473 heartbeater.cc:347] Connected to a master server at 127.21.143.126:42633
I20260619 18:46:27.085891 8473 heartbeater.cc:464] Registering TS with master...
I20260619 18:46:27.086391 8473 heartbeater.cc:511] Master 127.21.143.126:42633 requested a full tablet report, sending...
I20260619 18:46:27.088088 8285 ts_manager.cc:194] Registered new tserver with Master: 09fdbce739984769ad6edf4e85bf30ca (127.21.143.65:45485)
I20260619 18:46:27.089823 8285 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.143.65:50713
W20260619 18:46:27.190990 8476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:27.191241 8476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:27.191280 8476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:27.196018 8476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:27.196076 8476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:27.196132 8476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.66
I20260619 18:46:27.201022 8476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:46619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:42633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.66
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:27.202069 8476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:27.203452 8476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:27.212822 8484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:27.212822 8482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:27.212847 8481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:27.213112 8476 server_base.cc:1034] running on GCE node
I20260619 18:46:27.213521 8476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:27.214133 8476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:27.215293 8476 hybrid_clock.cc:648] HybridClock initialized: now 1781894787215280 us; error 24 us; skew 500 ppm
I20260619 18:46:27.217248 8476 webserver.cc:460] Webserver started at http://127.21.143.66:45791/ using document root <none> and password file <none>
I20260619 18:46:27.217844 8476 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:27.217906 8476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:27.218166 8476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:27.220202 8476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-1/data/instance:
uuid: "4c5857330b6b4745901ca2535aae3dd1"
format_stamp: "Formatted at 2026-06-19 18:46:27 on dist-test-slave-mld5"
server_key: "d97f1f485f172d3b0994c3b7ebd2ebe0"
server_key_iv: "43f558309ad784e5bc8ef8b29901c58c"
server_key_version: "encryptionkey@0"
I20260619 18:46:27.220690 8476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance:
uuid: "4c5857330b6b4745901ca2535aae3dd1"
format_stamp: "Formatted at 2026-06-19 18:46:27 on dist-test-slave-mld5"
server_key: "d97f1f485f172d3b0994c3b7ebd2ebe0"
server_key_iv: "43f558309ad784e5bc8ef8b29901c58c"
server_key_version: "encryptionkey@0"
I20260619 18:46:27.224213 8476 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:46:27.226540 8490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:27.227816 8476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:46:27.227931 8476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-1/wal
uuid: "4c5857330b6b4745901ca2535aae3dd1"
format_stamp: "Formatted at 2026-06-19 18:46:27 on dist-test-slave-mld5"
server_key: "d97f1f485f172d3b0994c3b7ebd2ebe0"
server_key_iv: "43f558309ad784e5bc8ef8b29901c58c"
server_key_version: "encryptionkey@0"
I20260619 18:46:27.228042 8476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:27.239135 8476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:27.239773 8476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:27.239956 8476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:27.240487 8476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:46:27.241362 8476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:46:27.241411 8476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:27.241492 8476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:46:27.241569 8476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:27.251978 8476 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.66:36975
I20260619 18:46:27.251991 8603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.66:36975 every 8 connection(s)
I20260619 18:46:27.252908 8476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
I20260619 18:46:27.257570 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 8476
I20260619 18:46:27.257706 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance
I20260619 18:46:27.258018 22077 external_mini_cluster.cc:1428] Setting key f3553562753d071123bee99dc1f8c1ca
I20260619 18:46:27.260076 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.67:0
--local_ip_for_outbound_sockets=127.21.143.67
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:42633
--builtin_ntp_servers=127.21.143.84:46619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 18:46:27.263375 8324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:27.254744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:48731 (local address 127.21.143.126:42633)
0619 18:46:27.254922 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:27.254926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:27.256061 (+ 1135us) server_negotiation.cc:407] Connection header received
0619 18:46:27.257569 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:27.257572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:27.257620 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:27.257723 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:27.258891 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:27.259639 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:27.261334 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:27.261632 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:27.261748 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:27.262853 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:27.262856 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:27.262859 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:27.262872 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:27.262891 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:27.262907 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:27.262989 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:27.263146 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 18:46:27.263208 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 18:46:27.264288 8606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:27.255205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42633 (local address 127.21.143.66:48731)
0619 18:46:27.255911 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:27.255951 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:27.257313 (+ 1362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:27.257931 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:27.257941 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:27.258063 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:27.258751 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:27.258777 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:27.259760 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:27.259763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:27.261213 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:27.261227 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:27.261785 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:27.261789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:27.261956 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:27.262599 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:46:27.262601 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:27.262640 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:46:27.262701 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:27.263002 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:27.263014 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:27.263023 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 18:46:27.263145 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 18:46:27.263456 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":130,"threads_started":1}
I20260619 18:46:27.265450 8604 heartbeater.cc:347] Connected to a master server at 127.21.143.126:42633
I20260619 18:46:27.265724 8604 heartbeater.cc:464] Registering TS with master...
I20260619 18:46:27.266391 8604 heartbeater.cc:511] Master 127.21.143.126:42633 requested a full tablet report, sending...
I20260619 18:46:27.267570 8285 ts_manager.cc:194] Registered new tserver with Master: 4c5857330b6b4745901ca2535aae3dd1 (127.21.143.66:36975)
I20260619 18:46:27.268142 8285 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.143.66:48731
W20260619 18:46:27.371289 8607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:27.371624 8607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:27.371695 8607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:27.375797 8607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:27.375851 8607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:27.375936 8607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.67
I20260619 18:46:27.380746 8607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:46619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:42633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.67
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:27.381820 8607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:27.383178 8607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:27.392536 8613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:27.392536 8615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:27.392599 8607 server_base.cc:1034] running on GCE node
W20260619 18:46:27.392532 8612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:27.393303 8607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:27.393900 8607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:27.395077 8607 hybrid_clock.cc:648] HybridClock initialized: now 1781894787395061 us; error 37 us; skew 500 ppm
I20260619 18:46:27.396842 8607 webserver.cc:460] Webserver started at http://127.21.143.67:36525/ using document root <none> and password file <none>
I20260619 18:46:27.397423 8607 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:27.397512 8607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:27.397720 8607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:27.399777 8607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-2/data/instance:
uuid: "6842ec5f67f24d85adc84a170d9d0496"
format_stamp: "Formatted at 2026-06-19 18:46:27 on dist-test-slave-mld5"
server_key: "b75f9128e6067696d750ca58daa4d099"
server_key_iv: "7622964468a5882b60348ca3a3f93c7b"
server_key_version: "encryptionkey@0"
I20260619 18:46:27.400328 8607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance:
uuid: "6842ec5f67f24d85adc84a170d9d0496"
format_stamp: "Formatted at 2026-06-19 18:46:27 on dist-test-slave-mld5"
server_key: "b75f9128e6067696d750ca58daa4d099"
server_key_iv: "7622964468a5882b60348ca3a3f93c7b"
server_key_version: "encryptionkey@0"
I20260619 18:46:27.403965 8607 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:46:27.406376 8621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:27.407603 8607 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:46:27.407728 8607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-2/wal
uuid: "6842ec5f67f24d85adc84a170d9d0496"
format_stamp: "Formatted at 2026-06-19 18:46:27 on dist-test-slave-mld5"
server_key: "b75f9128e6067696d750ca58daa4d099"
server_key_iv: "7622964468a5882b60348ca3a3f93c7b"
server_key_version: "encryptionkey@0"
I20260619 18:46:27.407840 8607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:27.435997 8607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:27.436704 8607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:27.436889 8607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:27.437491 8607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:46:27.438443 8607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:46:27.438515 8607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:27.438582 8607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:46:27.438635 8607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:27.447393 8607 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.67:33361
I20260619 18:46:27.447418 8734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.67:33361 every 8 connection(s)
I20260619 18:46:27.448261 8607 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
I20260619 18:46:27.456537 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 8607
I20260619 18:46:27.456650 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance
I20260619 18:46:27.456928 22077 external_mini_cluster.cc:1428] Setting key 9d75bb02cc2c5cbcfd7ae072f08efab3
I20260619 18:46:27.457758 8324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:27.449937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:36819 (local address 127.21.143.126:42633)
0619 18:46:27.450070 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:27.450074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:27.451312 (+ 1238us) server_negotiation.cc:407] Connection header received
0619 18:46:27.453057 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:27.453060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:27.453108 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:27.453180 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:27.454008 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:27.454519 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:27.455688 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:27.455898 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:27.455972 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:27.457198 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:27.457203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:27.457207 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:27.457221 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:27.457243 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:27.457253 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:27.457328 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:27.457547 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 18:46:27.457605 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 18:46:27.458747 8737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:27.450329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:42633 (local address 127.21.143.67:36819)
0619 18:46:27.451070 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:27.451170 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:27.452862 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:27.453331 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:27.453338 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:27.453427 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:27.453887 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:27.453899 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:27.454617 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:27.454620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:27.455588 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:27.455597 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:27.456015 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:27.456018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:27.456165 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:27.456863 (+ 698us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:46:27.456865 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:27.456903 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:46:27.456960 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:27.457357 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:27.457368 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:27.457376 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 18:46:27.457515 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 18:46:27.457849 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":110,"threads_started":1}
I20260619 18:46:27.459900 8735 heartbeater.cc:347] Connected to a master server at 127.21.143.126:42633
I20260619 18:46:27.460146 8735 heartbeater.cc:464] Registering TS with master...
I20260619 18:46:27.460623 8735 heartbeater.cc:511] Master 127.21.143.126:42633 requested a full tablet report, sending...
I20260619 18:46:27.461542 8285 ts_manager.cc:194] Registered new tserver with Master: 6842ec5f67f24d85adc84a170d9d0496 (127.21.143.67:33361)
I20260619 18:46:27.462103 8285 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.143.67:36819
I20260619 18:46:27.470619 22077 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:46:27.475860 8324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:27.472523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57818 (local address 127.21.143.126:42633)
0619 18:46:27.472631 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:27.472634 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:27.472759 (+ 125us) server_negotiation.cc:407] Connection header received
0619 18:46:27.472852 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:27.472854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:27.472888 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:27.472950 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:27.473168 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:27.473647 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:27.474378 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:27.474574 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:27.474642 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:27.475401 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:27.475404 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:27.475406 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:27.475416 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:27.475432 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:27.475439 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:27.475560 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:27.475728 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 18:46:27.475765 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 18:46:27.478422 8285 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:46:27.481436 8285 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:46:27.495890 8748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:27.490149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:36975 (local address 127.0.0.1:53012)
0619 18:46:27.491103 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:27.491117 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:27.491223 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:27.491774 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:27.491782 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:27.491819 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:46:27.491921 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:27.491925 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:27.492875 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:27.492880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:27.494258 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:27.494268 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:27.495242 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:27.495256 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:27.495404 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:27.495464 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 18:46:27.495567 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 18:46:27.495696 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"mutex_wait_us":227,"thread_start_us":89,"threads_started":1}
I20260619 18:46:27.496111 8750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:27.490514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50140 (local address 127.21.143.67:33361)
0619 18:46:27.490941 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:27.490953 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:27.490982 (+ 29us) server_negotiation.cc:407] Connection header received
0619 18:46:27.491217 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:27.491224 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:27.491462 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:27.491624 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:46:27.491991 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:27.493174 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:27.494824 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:27.495467 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:27.495580 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:27.495753 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:27.495805 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 18:46:27.495945 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":127,"threads_started":1}
I20260619 18:46:27.496212 8746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:27.489713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:45485 (local address 127.0.0.1:33184)
0619 18:46:27.490480 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:27.490547 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:27.490669 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:27.491667 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:27.491670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:27.491681 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:46:27.491903 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:27.491911 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:27.493308 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:27.493311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:27.494509 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:27.494519 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:27.495815 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:27.495819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:27.495897 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:27.495923 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 18:46:27.496004 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 18:46:27.496084 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":102,"thread_start_us":82,"threads_started":1}
I20260619 18:46:27.496510 8751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:27.490519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53012 (local address 127.21.143.66:36975)
0619 18:46:27.490976 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:27.490986 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:27.491225 (+ 239us) server_negotiation.cc:407] Connection header received
0619 18:46:27.491350 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:27.491355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:27.491508 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:27.491632 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:46:27.492046 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:27.492761 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:27.494381 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:27.495115 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:27.495224 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:27.495472 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:27.496230 (+ 758us) server_negotiation.cc:299] Negotiation successful
0619 18:46:27.496338 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":126,"threads_started":1}
I20260619 18:46:27.496748 8747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:27.489837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33184 (local address 127.21.143.65:45485)
0619 18:46:27.491201 (+ 1364us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:27.491211 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:27.491234 (+ 23us) server_negotiation.cc:407] Connection header received
0619 18:46:27.491358 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:27.491362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:27.491509 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:27.491832 (+ 323us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:46:27.491990 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:27.493174 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:27.494638 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:27.495453 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:27.495558 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:27.495733 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:27.496479 (+ 746us) server_negotiation.cc:299] Negotiation successful
0619 18:46:27.496577 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1246,"thread_start_us":133,"threads_started":1}
I20260619 18:46:27.497678 8749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:27.489897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:33361 (local address 127.0.0.1:50140)
0619 18:46:27.490896 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:27.490908 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:27.491041 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:27.491631 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:27.491639 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:27.491666 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:46:27.491855 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:27.491863 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:27.493308 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:27.493311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:27.494697 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:27.494704 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:27.495575 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:27.495579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:27.495667 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:27.495690 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 18:46:27.495754 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 18:46:27.495883 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":939,"mutex_wait_us":292,"spinlock_wait_cycles":1920,"thread_start_us":102,"threads_started":1}
I20260619 18:46:27.499182 8531 tablet_service.cc:1467] Processing CreateTablet for tablet 58fb85994dee4d9db588e85dc8590943 (DEFAULT_TABLE table=test-table [id=beabc11355c8470cae6f99caaa510d31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:46:27.499466 8407 tablet_service.cc:1467] Processing CreateTablet for tablet 58fb85994dee4d9db588e85dc8590943 (DEFAULT_TABLE table=test-table [id=beabc11355c8470cae6f99caaa510d31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:46:27.500240 8531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58fb85994dee4d9db588e85dc8590943. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:27.500492 8407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58fb85994dee4d9db588e85dc8590943. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:27.500245 8669 tablet_service.cc:1467] Processing CreateTablet for tablet 58fb85994dee4d9db588e85dc8590943 (DEFAULT_TABLE table=test-table [id=beabc11355c8470cae6f99caaa510d31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:46:27.501114 8669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58fb85994dee4d9db588e85dc8590943. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:27.506546 8752 tablet_bootstrap.cc:492] T 58fb85994dee4d9db588e85dc8590943 P 4c5857330b6b4745901ca2535aae3dd1: Bootstrap starting.
I20260619 18:46:27.509027 8752 tablet_bootstrap.cc:654] T 58fb85994dee4d9db588e85dc8590943 P 4c5857330b6b4745901ca2535aae3dd1: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:27.509778 8752 log.cc:824] T 58fb85994dee4d9db588e85dc8590943 P 4c5857330b6b4745901ca2535aae3dd1: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:27.510383 8753 tablet_bootstrap.cc:492] T 58fb85994dee4d9db588e85dc8590943 P 09fdbce739984769ad6edf4e85bf30ca: Bootstrap starting.
I20260619 18:46:27.511591 8754 tablet_bootstrap.cc:492] T 58fb85994dee4d9db588e85dc8590943 P 6842ec5f67f24d85adc84a170d9d0496: Bootstrap starting.
I20260619 18:46:27.511808 8752 tablet_bootstrap.cc:492] T 58fb85994dee4d9db588e85dc8590943 P 4c5857330b6b4745901ca2535aae3dd1: No bootstrap required, opened a new log
I20260619 18:46:27.512006 8752 ts_tablet_manager.cc:1397] T 58fb85994dee4d9db588e85dc8590943 P 4c5857330b6b4745901ca2535aae3dd1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 18:46:27.512591 8753 tablet_bootstrap.cc:654] T 58fb85994dee4d9db588e85dc8590943 P 09fdbce739984769ad6edf4e85bf30ca: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:27.513396 8753 log.cc:824] T 58fb85994dee4d9db588e85dc8590943 P 09fdbce739984769ad6edf4e85bf30ca: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:27.513649 8754 tablet_bootstrap.cc:654] T 58fb85994dee4d9db588e85dc8590943 P 6842ec5f67f24d85adc84a170d9d0496: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:27.514343 8754 log.cc:824] T 58fb85994dee4d9db588e85dc8590943 P 6842ec5f67f24d85adc84a170d9d0496: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:27.515372 8753 tablet_bootstrap.cc:492] T 58fb85994dee4d9db588e85dc8590943 P 09fdbce739984769ad6edf4e85bf30ca: No bootstrap required, opened a new log
I20260619 18:46:27.515551 8753 ts_tablet_manager.cc:1397] T 58fb85994dee4d9db588e85dc8590943 P 09fdbce739984769ad6edf4e85bf30ca: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260619 18:46:27.516296 8754 tablet_bootstrap.cc:492] T 58fb85994dee4d9db588e85dc8590943 P 6842ec5f67f24d85adc84a170d9d0496: No bootstrap required, opened a new log
I20260619 18:46:27.516460 8754 ts_tablet_manager.cc:1397] T 58fb85994dee4d9db588e85dc8590943 P 6842ec5f67f24d85adc84a170d9d0496: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 18:46:27.516376 8752 raft_consensus.cc:348] T 58fb85994dee4d9db588e85dc8590943 P 4c5857330b6b4745901ca2535aae3dd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09fdbce739984769ad6edf4e85bf30ca" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45485 } } peers { permanent_uuid: "6842ec5f67f24d85adc84a170d9d0496" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33361 } } peers { permanent_uuid: "4c5857330b6b4745901ca2535aae3dd1" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 36975 } }
I20260619 18:46:27.516665 8752 raft_consensus.cc:374] T 58fb85994dee4d9db588e85dc8590943 P 4c5857330b6b4745901ca2535aae3dd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:27.516721 8752 raft_consensus.cc:729] T 58fb85994dee4d9db588e85dc8590943 P 4c5857330b6b4745901ca2535aae3dd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c5857330b6b4745901ca2535aae3dd1, State: Initialized, Role: FOLLOWER
I20260619 18:46:27.517177 8752 consensus_queue.cc:260] T 58fb85994dee4d9db588e85dc8590943 P 4c5857330b6b4745901ca2535aae3dd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09fdbce739984769ad6edf4e85bf30ca" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45485 } } peers { permanent_uuid: "6842ec5f67f24d85adc84a170d9d0496" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33361 } } peers { permanent_uuid: "4c5857330b6b4745901ca2535aae3dd1" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 36975 } }
I20260619 18:46:27.517836 8604 heartbeater.cc:503] Master 127.21.143.126:42633 was elected leader, sending a full tablet report...
I20260619 18:46:27.518195 8752 ts_tablet_manager.cc:1428] T 58fb85994dee4d9db588e85dc8590943 P 4c5857330b6b4745901ca2535aae3dd1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 18:46:27.520092 8753 raft_consensus.cc:348] T 58fb85994dee4d9db588e85dc8590943 P 09fdbce739984769ad6edf4e85bf30ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09fdbce739984769ad6edf4e85bf30ca" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45485 } } peers { permanent_uuid: "6842ec5f67f24d85adc84a170d9d0496" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33361 } } peers { permanent_uuid: "4c5857330b6b4745901ca2535aae3dd1" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 36975 } }
I20260619 18:46:27.520344 8753 raft_consensus.cc:374] T 58fb85994dee4d9db588e85dc8590943 P 09fdbce739984769ad6edf4e85bf30ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:27.520399 8753 raft_consensus.cc:729] T 58fb85994dee4d9db588e85dc8590943 P 09fdbce739984769ad6edf4e85bf30ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09fdbce739984769ad6edf4e85bf30ca, State: Initialized, Role: FOLLOWER
I20260619 18:46:27.520812 8753 consensus_queue.cc:260] T 58fb85994dee4d9db588e85dc8590943 P 09fdbce739984769ad6edf4e85bf30ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09fdbce739984769ad6edf4e85bf30ca" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45485 } } peers { permanent_uuid: "6842ec5f67f24d85adc84a170d9d0496" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33361 } } peers { permanent_uuid: "4c5857330b6b4745901ca2535aae3dd1" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 36975 } }
I20260619 18:46:27.520721 8754 raft_consensus.cc:348] T 58fb85994dee4d9db588e85dc8590943 P 6842ec5f67f24d85adc84a170d9d0496 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09fdbce739984769ad6edf4e85bf30ca" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45485 } } peers { permanent_uuid: "6842ec5f67f24d85adc84a170d9d0496" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33361 } } peers { permanent_uuid: "4c5857330b6b4745901ca2535aae3dd1" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 36975 } }
I20260619 18:46:27.520944 8754 raft_consensus.cc:374] T 58fb85994dee4d9db588e85dc8590943 P 6842ec5f67f24d85adc84a170d9d0496 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:27.521003 8754 raft_consensus.cc:729] T 58fb85994dee4d9db588e85dc8590943 P 6842ec5f67f24d85adc84a170d9d0496 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6842ec5f67f24d85adc84a170d9d0496, State: Initialized, Role: FOLLOWER
I20260619 18:46:27.521430 8754 consensus_queue.cc:260] T 58fb85994dee4d9db588e85dc8590943 P 6842ec5f67f24d85adc84a170d9d0496 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09fdbce739984769ad6edf4e85bf30ca" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45485 } } peers { permanent_uuid: "6842ec5f67f24d85adc84a170d9d0496" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33361 } } peers { permanent_uuid: "4c5857330b6b4745901ca2535aae3dd1" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 36975 } }
I20260619 18:46:27.521731 8753 ts_tablet_manager.cc:1428] T 58fb85994dee4d9db588e85dc8590943 P 09fdbce739984769ad6edf4e85bf30ca: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260619 18:46:27.522034 8473 heartbeater.cc:503] Master 127.21.143.126:42633 was elected leader, sending a full tablet report...
I20260619 18:46:27.522109 8735 heartbeater.cc:503] Master 127.21.143.126:42633 was elected leader, sending a full tablet report...
I20260619 18:46:27.523115 8754 ts_tablet_manager.cc:1428] T 58fb85994dee4d9db588e85dc8590943 P 6842ec5f67f24d85adc84a170d9d0496: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260619 18:46:27.574879 8474 tablet.cc:2369] T 58fb85994dee4d9db588e85dc8590943 P 09fdbce739984769ad6edf4e85bf30ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:46:27.604768 8760 raft_consensus.cc:482] T 58fb85994dee4d9db588e85dc8590943 P 6842ec5f67f24d85adc84a170d9d0496 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:46:27.604964 8760 raft_consensus.cc:504] T 58fb85994dee4d9db588e85dc8590943 P 6842ec5f67f24d85adc84a170d9d0496 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09fdbce739984769ad6edf4e85bf30ca" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45485 } } peers { permanent_uuid: "6842ec5f67f24d85adc84a170d9d0496" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33361 } } peers { permanent_uuid: "4c5857330b6b4745901ca2535aae3dd1" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 36975 } }
I20260619 18:46:27.606097 8760 leader_election.cc:288] T 58fb85994dee4d9db588e85dc8590943 P 6842ec5f67f24d85adc84a170d9d0496 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09fdbce739984769ad6edf4e85bf30ca (127.21.143.65:45485), 4c5857330b6b4745901ca2535aae3dd1 (127.21.143.66:36975)
I20260619 18:46:27.609733 8747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:27.606516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:35709 (local address 127.21.143.65:45485)
0619 18:46:27.606670 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:27.606675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:27.606697 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:46:27.606746 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:27.606747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:27.606787 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:27.606883 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:46:27.607160 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:27.607716 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:27.608801 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:27.609197 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:27.609326 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:27.609376 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:27.609560 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 18:46:27.609623 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 18:46:27.609733 8737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:27.606282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:45485 (local address 127.21.143.67:35709)
0619 18:46:27.606417 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:27.606433 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:27.606527 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:27.606896 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:27.606899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:27.606917 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:46:27.607039 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:27.607045 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:27.607821 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:27.607827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:27.608679 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:27.608695 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:27.609315 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:27.609318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:27.609413 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:27.609434 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 18:46:27.609535 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 18:46:27.609605 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 18:46:27.610106 8761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:27.606396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:36975 (local address 127.21.143.67:59369)
0619 18:46:27.606766 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:27.606776 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:27.606910 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:27.607167 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:27.607170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:27.607185 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:46:27.607329 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:27.607334 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:27.608016 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:27.608022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:27.608932 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:27.608941 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:27.609652 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:27.609656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:27.609751 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:27.609777 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 18:46:27.609923 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 18:46:27.609993 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":135,"threads_started":1}
I20260619 18:46:27.610148 8751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:27.606503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:59369 (local address 127.21.143.66:36975)
0619 18:46:27.606637 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:27.606642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:27.606858 (+ 216us) server_negotiation.cc:407] Connection header received
0619 18:46:27.607028 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:27.607030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:27.607070 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:27.607147 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:46:27.607438 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:27.607913 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:27.609047 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:27.609520 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:27.609655 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:27.609699 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:27.609956 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 18:46:27.610031 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 18:46:27.610680 8427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58fb85994dee4d9db588e85dc8590943" candidate_uuid: "6842ec5f67f24d85adc84a170d9d0496" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09fdbce739984769ad6edf4e85bf30ca" is_pre_election: true
I20260619 18:46:27.610687 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58fb85994dee4d9db588e85dc8590943" candidate_uuid: "6842ec5f67f24d85adc84a170d9d0496" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c5857330b6b4745901ca2535aae3dd1" is_pre_election: true
I20260619 18:46:27.610984 8558 raft_consensus.cc:2454] T 58fb85994dee4d9db588e85dc8590943 P 4c5857330b6b4745901ca2535aae3dd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6842ec5f67f24d85adc84a170d9d0496 in term 0.
I20260619 18:46:27.610973 8427 raft_consensus.cc:2454] T 58fb85994dee4d9db588e85dc8590943 P 09fdbce739984769ad6edf4e85bf30ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6842ec5f67f24d85adc84a170d9d0496 in term 0.
I20260619 18:46:27.611493 8625 leader_election.cc:302] T 58fb85994dee4d9db588e85dc8590943 P 6842ec5f67f24d85adc84a170d9d0496 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09fdbce739984769ad6edf4e85bf30ca, 6842ec5f67f24d85adc84a170d9d0496; no voters:
I20260619 18:46:27.611778 8760 raft_consensus.cc:2781] T 58fb85994dee4d9db588e85dc8590943 P 6842ec5f67f24d85adc84a170d9d0496 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:46:27.611857 8760 raft_consensus.cc:482] T 58fb85994dee4d9db588e85dc8590943 P 6842ec5f67f24d85adc84a170d9d0496 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:46:27.611905 8760 raft_consensus.cc:3037] T 58fb85994dee4d9db588e85dc8590943 P 6842ec5f67f24d85adc84a170d9d0496 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:27.612862 8760 raft_consensus.cc:504] T 58fb85994dee4d9db588e85dc8590943 P 6842ec5f67f24d85adc84a170d9d0496 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09fdbce739984769ad6edf4e85bf30ca" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45485 } } peers { permanent_uuid: "6842ec5f67f24d85adc84a170d9d0496" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33361 } } peers { permanent_uuid: "4c5857330b6b4745901ca2535aae3dd1" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 36975 } }
I20260619 18:46:27.613240 8760 leader_election.cc:288] T 58fb85994dee4d9db588e85dc8590943 P 6842ec5f67f24d85adc84a170d9d0496 [CANDIDATE]: Term 1 election: Requested vote from peers 09fdbce739984769ad6edf4e85bf30ca (127.21.143.65:45485), 4c5857330b6b4745901ca2535aae3dd1 (127.21.143.66:36975)
I20260619 18:46:27.613561 8427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58fb85994dee4d9db588e85dc8590943" candidate_uuid: "6842ec5f67f24d85adc84a170d9d0496" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09fdbce739984769ad6edf4e85bf30ca"
I20260619 18:46:27.613616 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58fb85994dee4d9db588e85dc8590943" candidate_uuid: "6842ec5f67f24d85adc84a170d9d0496" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c5857330b6b4745901ca2535aae3dd1"
I20260619 18:46:27.613674 8427 raft_consensus.cc:3037] T 58fb85994dee4d9db588e85dc8590943 P 09fdbce739984769ad6edf4e85bf30ca [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:27.613770 8558 raft_consensus.cc:3037] T 58fb85994dee4d9db588e85dc8590943 P 4c5857330b6b4745901ca2535aae3dd1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:27.614715 8427 raft_consensus.cc:2454] T 58fb85994dee4d9db588e85dc8590943 P 09fdbce739984769ad6edf4e85bf30ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6842ec5f67f24d85adc84a170d9d0496 in term 1.
I20260619 18:46:27.614881 8558 raft_consensus.cc:2454] T 58fb85994dee4d9db588e85dc8590943 P 4c5857330b6b4745901ca2535aae3dd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6842ec5f67f24d85adc84a170d9d0496 in term 1.
I20260619 18:46:27.615046 8625 leader_election.cc:302] T 58fb85994dee4d9db588e85dc8590943 P 6842ec5f67f24d85adc84a170d9d0496 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09fdbce739984769ad6edf4e85bf30ca, 6842ec5f67f24d85adc84a170d9d0496; no voters:
I20260619 18:46:27.615223 8760 raft_consensus.cc:2781] T 58fb85994dee4d9db588e85dc8590943 P 6842ec5f67f24d85adc84a170d9d0496 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:27.615463 8760 raft_consensus.cc:686] T 58fb85994dee4d9db588e85dc8590943 P 6842ec5f67f24d85adc84a170d9d0496 [term 1 LEADER]: Becoming Leader. State: Replica: 6842ec5f67f24d85adc84a170d9d0496, State: Running, Role: LEADER
I20260619 18:46:27.615811 8760 consensus_queue.cc:237] T 58fb85994dee4d9db588e85dc8590943 P 6842ec5f67f24d85adc84a170d9d0496 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09fdbce739984769ad6edf4e85bf30ca" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45485 } } peers { permanent_uuid: "6842ec5f67f24d85adc84a170d9d0496" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33361 } } peers { permanent_uuid: "4c5857330b6b4745901ca2535aae3dd1" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 36975 } }
I20260619 18:46:27.619006 8280 catalog_manager.cc:5495] T 58fb85994dee4d9db588e85dc8590943 P 6842ec5f67f24d85adc84a170d9d0496 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6842ec5f67f24d85adc84a170d9d0496 (127.21.143.67). New cstate: current_term: 1 leader_uuid: "6842ec5f67f24d85adc84a170d9d0496" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09fdbce739984769ad6edf4e85bf30ca" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6842ec5f67f24d85adc84a170d9d0496" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 33361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c5857330b6b4745901ca2535aae3dd1" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 36975 } health_report { overall_health: UNKNOWN } } }
I20260619 18:46:27.654587 8750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:27.650961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50146 (local address 127.21.143.67:33361)
0619 18:46:27.651111 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:27.651115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:27.651131 (+ 16us) server_negotiation.cc:407] Connection header received
0619 18:46:27.651276 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:27.651278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:27.651388 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:27.651493 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:46:27.651757 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:27.652272 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:27.653147 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:27.653364 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:27.653474 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:27.653769 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:46:27.654154 (+ 385us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:46:27.654309 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:27.654399 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 18:46:27.654433 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 18:46:27.660598 8427 raft_consensus.cc:1261] T 58fb85994dee4d9db588e85dc8590943 P 09fdbce739984769ad6edf4e85bf30ca [term 1 FOLLOWER]: Refusing update from remote peer 6842ec5f67f24d85adc84a170d9d0496: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:46:27.660597 8558 raft_consensus.cc:1261] T 58fb85994dee4d9db588e85dc8590943 P 4c5857330b6b4745901ca2535aae3dd1 [term 1 FOLLOWER]: Refusing update from remote peer 6842ec5f67f24d85adc84a170d9d0496: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:46:27.661302 8760 consensus_queue.cc:1035] T 58fb85994dee4d9db588e85dc8590943 P 6842ec5f67f24d85adc84a170d9d0496 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09fdbce739984769ad6edf4e85bf30ca" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 45485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:46:27.661506 8762 consensus_queue.cc:1035] T 58fb85994dee4d9db588e85dc8590943 P 6842ec5f67f24d85adc84a170d9d0496 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c5857330b6b4745901ca2535aae3dd1" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 36975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:46:27.668741 8765 mvcc.cc:205] Tried to move back new op lower bound from 7298641050250575872 to 7298641050077188096. Current Snapshot: MvccSnapshot[applied={T|T < 7298641050250575872}]
I20260619 18:46:27.676400 8280 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:46:27.676582 8280 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:46:27.679178 8280 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d37ae677ab14bbaaefb58212b5c5e49: Sending DeleteTablet for 3 replicas of tablet 58fb85994dee4d9db588e85dc8590943
I20260619 18:46:27.679953 8531 tablet_service.cc:1514] Processing DeleteTablet for tablet 58fb85994dee4d9db588e85dc8590943 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:46:27 UTC) from {username='slave'} at 127.0.0.1:53012
I20260619 18:46:27.680025 8669 tablet_service.cc:1514] Processing DeleteTablet for tablet 58fb85994dee4d9db588e85dc8590943 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:46:27 UTC) from {username='slave'} at 127.0.0.1:50140
I20260619 18:46:27.680336 8537 tablet_service.cc:1514] Processing DeleteTablet for tablet 58fb85994dee4d9db588e85dc8590943 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:46:27 UTC) from {username='slave'} at 127.0.0.1:53012
I20260619 18:46:27.680449 8775 tablet_replica.cc:330] T 58fb85994dee4d9db588e85dc8590943 P 4c5857330b6b4745901ca2535aae3dd1: stopping tablet replica
I20260619 18:46:27.680574 8776 tablet_replica.cc:330] T 58fb85994dee4d9db588e85dc8590943 P 6842ec5f67f24d85adc84a170d9d0496: stopping tablet replica
I20260619 18:46:27.680835 8775 raft_consensus.cc:2229] T 58fb85994dee4d9db588e85dc8590943 P 4c5857330b6b4745901ca2535aae3dd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:46:27.680843 8776 raft_consensus.cc:2229] T 58fb85994dee4d9db588e85dc8590943 P 6842ec5f67f24d85adc84a170d9d0496 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:46:27.680917 8407 tablet_service.cc:1514] Processing DeleteTablet for tablet 58fb85994dee4d9db588e85dc8590943 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:46:27 UTC) from {username='slave'} at 127.0.0.1:33184
I20260619 18:46:27.681149 8776 raft_consensus.cc:2258] T 58fb85994dee4d9db588e85dc8590943 P 6842ec5f67f24d85adc84a170d9d0496 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:46:27.681301 8775 raft_consensus.cc:2258] T 58fb85994dee4d9db588e85dc8590943 P 4c5857330b6b4745901ca2535aae3dd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:46:27.681480 8777 tablet_replica.cc:330] T 58fb85994dee4d9db588e85dc8590943 P 09fdbce739984769ad6edf4e85bf30ca: stopping tablet replica
I20260619 18:46:27.682286 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 8345
I20260619 18:46:27.682859 8776 ts_tablet_manager.cc:1905] T 58fb85994dee4d9db588e85dc8590943 P 6842ec5f67f24d85adc84a170d9d0496: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:46:27.682859 8775 ts_tablet_manager.cc:1905] T 58fb85994dee4d9db588e85dc8590943 P 4c5857330b6b4745901ca2535aae3dd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:46:27.685595 8775 ts_tablet_manager.cc:1918] T 58fb85994dee4d9db588e85dc8590943 P 4c5857330b6b4745901ca2535aae3dd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:46:27.685595 8776 ts_tablet_manager.cc:1918] T 58fb85994dee4d9db588e85dc8590943 P 6842ec5f67f24d85adc84a170d9d0496: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:46:27.685688 8776 log.cc:1196] T 58fb85994dee4d9db588e85dc8590943 P 6842ec5f67f24d85adc84a170d9d0496: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-2/wal/wals/58fb85994dee4d9db588e85dc8590943
I20260619 18:46:27.685799 8775 log.cc:1196] T 58fb85994dee4d9db588e85dc8590943 P 4c5857330b6b4745901ca2535aae3dd1: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894542572984-22077-0/minicluster-data/ts-1/wal/wals/58fb85994dee4d9db588e85dc8590943
I20260619 18:46:27.686036 8776 ts_tablet_manager.cc:1939] T 58fb85994dee4d9db588e85dc8590943 P 6842ec5f67f24d85adc84a170d9d0496: Deleting consensus metadata
I20260619 18:46:27.686225 8775 ts_tablet_manager.cc:1939] T 58fb85994dee4d9db588e85dc8590943 P 4c5857330b6b4745901ca2535aae3dd1: Deleting consensus metadata
I20260619 18:46:27.687258 8270 catalog_manager.cc:4841] TS 4c5857330b6b4745901ca2535aae3dd1 (127.21.143.66:36975): tablet 58fb85994dee4d9db588e85dc8590943 (table test-table [id=beabc11355c8470cae6f99caaa510d31]) successfully deleted
W20260619 18:46:27.687969 8270 catalog_manager.cc:4808] TS 4c5857330b6b4745901ca2535aae3dd1 (127.21.143.66:36975): delete failed for tablet 58fb85994dee4d9db588e85dc8590943 because the tablet was not found. No further retry: Not found: Tablet not found: 58fb85994dee4d9db588e85dc8590943
I20260619 18:46:27.688306 8271 catalog_manager.cc:4841] TS 6842ec5f67f24d85adc84a170d9d0496 (127.21.143.67:33361): tablet 58fb85994dee4d9db588e85dc8590943 (table test-table [id=beabc11355c8470cae6f99caaa510d31]) successfully deleted
W20260619 18:46:27.693835 8272 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.143.65:45485 (error 108)
I20260619 18:46:27.694684 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 8476
I20260619 18:46:27.694905 8749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:27.694591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:45485 (local address 127.0.0.1:33194)
0619 18:46:27.694740 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:27.694824 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.65:45485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 18:46:27.696273 8272 catalog_manager.cc:4584] TS 09fdbce739984769ad6edf4e85bf30ca (127.21.143.65:45485): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 58fb85994dee4d9db588e85dc8590943: Network error: Client connection negotiation failed: client connection to 127.21.143.65:45485: connect: Connection refused (error 111)
I20260619 18:46:27.701989 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 8607
I20260619 18:46:27.707995 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 8254
2026-06-19T18:46:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1068 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:27 dist-test-slave-mld5 krb5kdc[8781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:27 dist-test-slave-mld5 krb5kdc[8781](info): set up 2 sockets
Jun 19 18:46:27 dist-test-slave-mld5 krb5kdc[8781](info): commencing operation
krb5kdc: starting...
W20260619 18:46:28.739174 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:46:28 dist-test-slave-mld5 krb5kdc[8781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.143.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
I20260619 18:46:28.856694 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:36101
--webserver_interface=127.21.143.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:34755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:36101
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:28.968173 8794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:28.968499 8794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:28.968582 8794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:28.973429 8794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:28.973526 8794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:28.973562 8794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:28.973600 8794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:28.973635 8794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:28.978991 8794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:34755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:36101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:36101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:28.980345 8794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:28.981402 8794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:28.989665 8800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:28.989807 8794 server_base.cc:1034] running on GCE node
W20260619 18:46:28.989655 8802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:28.989655 8799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:28.990554 8794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:28.991753 8794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:28.992935 8794 hybrid_clock.cc:648] HybridClock initialized: now 1781894788992917 us; error 32 us; skew 500 ppm
Jun 19 18:46:28 dist-test-slave-mld5 krb5kdc[8781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894788, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:28.995617 8794 init.cc:377] Logged in from keytab as kudu/127.21.143.126@KRBTEST.COM (short username kudu)
I20260619 18:46:28.996492 8794 webserver.cc:460] Webserver started at http://127.21.143.126:45925/ using document root <none> and password file <none>
I20260619 18:46:28.997069 8794 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:28.997159 8794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:28.997386 8794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:28.999459 8794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "ef059973c2354c0daafefb707aa70d27"
format_stamp: "Formatted at 2026-06-19 18:46:28 on dist-test-slave-mld5"
server_key: "d0b37ee5e4c20fa44b513c94f44b5371"
server_key_iv: "85920a88c97c306a0a3ea276579232cb"
server_key_version: "encryptionkey@0"
I20260619 18:46:29.000018 8794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "ef059973c2354c0daafefb707aa70d27"
format_stamp: "Formatted at 2026-06-19 18:46:28 on dist-test-slave-mld5"
server_key: "d0b37ee5e4c20fa44b513c94f44b5371"
server_key_iv: "85920a88c97c306a0a3ea276579232cb"
server_key_version: "encryptionkey@0"
I20260619 18:46:29.003708 8794 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:46:29.006028 8809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:29.007181 8794 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:46:29.007362 8794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "ef059973c2354c0daafefb707aa70d27"
format_stamp: "Formatted at 2026-06-19 18:46:28 on dist-test-slave-mld5"
server_key: "d0b37ee5e4c20fa44b513c94f44b5371"
server_key_iv: "85920a88c97c306a0a3ea276579232cb"
server_key_version: "encryptionkey@0"
I20260619 18:46:29.007474 8794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:29.048321 8794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:29.051116 8794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:29.051379 8794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:29.058513 8794 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:36101
I20260619 18:46:29.058619 8861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:36101 every 8 connection(s)
I20260619 18:46:29.059518 8794 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:29.062112 8862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:29.063139 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 8794
I20260619 18:46:29.063242 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:46:29.063593 22077 external_mini_cluster.cc:1428] Setting key fa9954cfcee8258e617b16bede61795b
I20260619 18:46:29.068152 8862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef059973c2354c0daafefb707aa70d27: Bootstrap starting.
I20260619 18:46:29.070750 8862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef059973c2354c0daafefb707aa70d27: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:29.071712 8862 log.cc:824] T 00000000000000000000000000000000 P ef059973c2354c0daafefb707aa70d27: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:46:29 dist-test-slave-mld5 krb5kdc[8781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:46:29.073750 8862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef059973c2354c0daafefb707aa70d27: No bootstrap required, opened a new log
I20260619 18:46:29.077601 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:29.064931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42020 (local address 127.21.143.126:36101)
0619 18:46:29.065291 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:29.065315 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:29.065349 (+ 34us) server_negotiation.cc:407] Connection header received
0619 18:46:29.065951 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:29.065967 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:29.066271 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:29.066529 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:29.067432 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.068438 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.069547 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.069993 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.070104 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:29.073123 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:29.073127 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:29.073140 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:29.073165 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:29.074907 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:29.075408 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:29.075412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:29.075416 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:29.075518 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:29.075810 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:29.075812 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:29.075813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:29.076162 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:29.076312 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:29.076586 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 18:46:29.076900 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":113,"threads_started":1}
I20260619 18:46:29.077589 8862 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef059973c2354c0daafefb707aa70d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef059973c2354c0daafefb707aa70d27" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 36101 } }
I20260619 18:46:29.077873 8862 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef059973c2354c0daafefb707aa70d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:29.077961 8862 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef059973c2354c0daafefb707aa70d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef059973c2354c0daafefb707aa70d27, State: Initialized, Role: FOLLOWER
I20260619 18:46:29.078428 8862 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef059973c2354c0daafefb707aa70d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef059973c2354c0daafefb707aa70d27" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 36101 } }
I20260619 18:46:29.078576 8862 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef059973c2354c0daafefb707aa70d27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:29.078645 8862 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef059973c2354c0daafefb707aa70d27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:29.078768 8862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef059973c2354c0daafefb707aa70d27 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:29.079738 8862 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef059973c2354c0daafefb707aa70d27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef059973c2354c0daafefb707aa70d27" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 36101 } }
I20260619 18:46:29.080067 8862 leader_election.cc:302] T 00000000000000000000000000000000 P ef059973c2354c0daafefb707aa70d27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef059973c2354c0daafefb707aa70d27; no voters:
I20260619 18:46:29.080308 8862 leader_election.cc:288] T 00000000000000000000000000000000 P ef059973c2354c0daafefb707aa70d27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:29.080430 8867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef059973c2354c0daafefb707aa70d27 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:29.080662 8867 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef059973c2354c0daafefb707aa70d27 [term 1 LEADER]: Becoming Leader. State: Replica: ef059973c2354c0daafefb707aa70d27, State: Running, Role: LEADER
I20260619 18:46:29.081001 8867 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef059973c2354c0daafefb707aa70d27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef059973c2354c0daafefb707aa70d27" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 36101 } }
I20260619 18:46:29.081427 8862 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef059973c2354c0daafefb707aa70d27 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:29.082640 8869 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef059973c2354c0daafefb707aa70d27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef059973c2354c0daafefb707aa70d27. Latest consensus state: current_term: 1 leader_uuid: "ef059973c2354c0daafefb707aa70d27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef059973c2354c0daafefb707aa70d27" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 36101 } } }
I20260619 18:46:29.082649 8868 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef059973c2354c0daafefb707aa70d27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef059973c2354c0daafefb707aa70d27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef059973c2354c0daafefb707aa70d27" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 36101 } } }
I20260619 18:46:29.082764 8868 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef059973c2354c0daafefb707aa70d27 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:29.083159 8876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:29.082763 8869 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef059973c2354c0daafefb707aa70d27 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:29.085995 8876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:29.091102 8876 catalog_manager.cc:1347] Generated new cluster ID: 2d1d493045114b5199cd2e47d71834eb
I20260619 18:46:29.091176 8876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:29.111766 8876 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:29.112789 8876 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:29.121363 8876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef059973c2354c0daafefb707aa70d27: Generated new TSK 0
I20260619 18:46:29.122005 8876 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.143.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab.
Entry for principal kudu/127.21.143.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab.
I20260619 18:46:29.154790 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.65:0
--local_ip_for_outbound_sockets=127.21.143.65
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:36101
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:34755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:29.267093 8888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:29.267388 8888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:29.267434 8888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:29.271433 8888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:29.271481 8888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:29.271536 8888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.65
I20260619 18:46:29.276628 8888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:34755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.65.keytab
--principal=kudu/127.21.143.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:36101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.65
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:29.277524 8888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:29.278430 8888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:29.287703 8894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:29.287703 8896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:29.287782 8888 server_base.cc:1034] running on GCE node
W20260619 18:46:29.287719 8893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:29.288295 8888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:29.288852 8888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:29.290030 8888 hybrid_clock.cc:648] HybridClock initialized: now 1781894789290022 us; error 30 us; skew 500 ppm
Jun 19 18:46:29 dist-test-slave-mld5 krb5kdc[8781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894789, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:29.292712 8888 init.cc:377] Logged in from keytab as kudu/127.21.143.65@KRBTEST.COM (short username kudu)
I20260619 18:46:29.293558 8888 webserver.cc:460] Webserver started at http://127.21.143.65:40903/ using document root <none> and password file <none>
I20260619 18:46:29.294090 8888 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:29.294176 8888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:29.294399 8888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:29.296482 8888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-0/data/instance:
uuid: "2b502f45c71749cfbf7c7fc260f4dda8"
format_stamp: "Formatted at 2026-06-19 18:46:29 on dist-test-slave-mld5"
server_key: "ad1e090a186078270bb02fc047047cd3"
server_key_iv: "98f23eacabd299f5b184dbbb626e76fc"
server_key_version: "encryptionkey@0"
I20260619 18:46:29.297024 8888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance:
uuid: "2b502f45c71749cfbf7c7fc260f4dda8"
format_stamp: "Formatted at 2026-06-19 18:46:29 on dist-test-slave-mld5"
server_key: "ad1e090a186078270bb02fc047047cd3"
server_key_iv: "98f23eacabd299f5b184dbbb626e76fc"
server_key_version: "encryptionkey@0"
I20260619 18:46:29.300475 8888 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:46:29.302652 8903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:29.303891 8888 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:46:29.304020 8888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-0/wal
uuid: "2b502f45c71749cfbf7c7fc260f4dda8"
format_stamp: "Formatted at 2026-06-19 18:46:29 on dist-test-slave-mld5"
server_key: "ad1e090a186078270bb02fc047047cd3"
server_key_iv: "98f23eacabd299f5b184dbbb626e76fc"
server_key_version: "encryptionkey@0"
I20260619 18:46:29.304126 8888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:29.314328 8888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:29.317394 8888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:29.317581 8888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:29.318099 8888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:46:29.318847 8888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:46:29.318929 8888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:29.318997 8888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:46:29.319041 8888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:29.328080 8888 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.65:35889
I20260619 18:46:29.328110 9016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.65:35889 every 8 connection(s)
I20260619 18:46:29.328917 8888 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
I20260619 18:46:29.330842 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 8888
I20260619 18:46:29.330943 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance
I20260619 18:46:29.331183 22077 external_mini_cluster.cc:1428] Setting key 87342320324a520d219a05ea6d2e56f9
Jun 19 18:46:29 dist-test-slave-mld5 krb5kdc[8781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894789, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.65@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.143.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.66@KRBTEST.COM" created.
I20260619 18:46:29.344183 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:29.330509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:49919 (local address 127.21.143.126:36101)
0619 18:46:29.330640 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:29.330645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:29.331582 (+ 937us) server_negotiation.cc:407] Connection header received
0619 18:46:29.333204 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:29.333207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:29.333247 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:29.333326 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:29.334734 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.335607 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.337365 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.337579 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.337655 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:29.340409 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:29.340415 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:29.340422 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:29.340446 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:29.341886 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:29.342477 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:29.342481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:29.342483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:29.342566 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:29.342918 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:29.342922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:29.342924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:29.343112 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:29.343228 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:29.343873 (+ 645us) server_negotiation.cc:299] Negotiation successful
0619 18:46:29.344007 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 18:46:29.344977 9019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:29.330936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:36101 (local address 127.21.143.65:49919)
0619 18:46:29.331441 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:29.331492 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:29.332998 (+ 1506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:29.333528 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:29.333541 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:29.334018 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:29.334578 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:29.334591 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.335726 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.335733 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:29.337252 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:29.337262 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.337739 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.337745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:29.337991 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:29.338568 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:29.338594 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:29.340235 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:29.342005 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:29.342011 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:29.342025 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:29.342366 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:29.342669 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:29.342672 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:29.342673 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:29.342819 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:29.343261 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:29.343268 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:29.343605 (+ 337us) client_negotiation.cc:769] Sending connection context
0619 18:46:29.343817 (+ 212us) client_negotiation.cc:240] Negotiation successful
0619 18:46:29.344148 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":100,"threads_started":1}
I20260619 18:46:29.346366 9017 heartbeater.cc:347] Connected to a master server at 127.21.143.126:36101
I20260619 18:46:29.346626 9017 heartbeater.cc:464] Registering TS with master...
I20260619 18:46:29.347218 9017 heartbeater.cc:511] Master 127.21.143.126:36101 requested a full tablet report, sending...
I20260619 18:46:29.348829 8826 ts_manager.cc:194] Registered new tserver with Master: 2b502f45c71749cfbf7c7fc260f4dda8 (127.21.143.65:35889)
I20260619 18:46:29.349962 8826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.65@KRBTEST.COM'} at 127.21.143.65:49919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab.
Entry for principal kudu/127.21.143.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab.
I20260619 18:46:29.359865 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.66:0
--local_ip_for_outbound_sockets=127.21.143.66
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:36101
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:34755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:29.467486 9022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:29.467736 9022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:29.467779 9022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:29.471647 9022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:29.471696 9022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:29.471753 9022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.66
I20260619 18:46:29.476892 9022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:34755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.66.keytab
--principal=kudu/127.21.143.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:36101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.66
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:29.477818 9022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:29.478747 9022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:29.488195 9027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:29.488195 9028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:29.488240 9022 server_base.cc:1034] running on GCE node
W20260619 18:46:29.488219 9030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:29.488787 9022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:29.489429 9022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:29.490612 9022 hybrid_clock.cc:648] HybridClock initialized: now 1781894789490587 us; error 47 us; skew 500 ppm
Jun 19 18:46:29 dist-test-slave-mld5 krb5kdc[8781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894789, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:29.493489 9022 init.cc:377] Logged in from keytab as kudu/127.21.143.66@KRBTEST.COM (short username kudu)
I20260619 18:46:29.494380 9022 webserver.cc:460] Webserver started at http://127.21.143.66:32991/ using document root <none> and password file <none>
I20260619 18:46:29.495065 9022 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:29.495155 9022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:29.495445 9022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:29.497422 9022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-1/data/instance:
uuid: "58edcb95bc9c42a69106373fc359f0f1"
format_stamp: "Formatted at 2026-06-19 18:46:29 on dist-test-slave-mld5"
server_key: "4b4376dc83fac237204a9eaa9201520f"
server_key_iv: "be14279eb4d889cc6d29065c5e4d0b89"
server_key_version: "encryptionkey@0"
I20260619 18:46:29.497972 9022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance:
uuid: "58edcb95bc9c42a69106373fc359f0f1"
format_stamp: "Formatted at 2026-06-19 18:46:29 on dist-test-slave-mld5"
server_key: "4b4376dc83fac237204a9eaa9201520f"
server_key_iv: "be14279eb4d889cc6d29065c5e4d0b89"
server_key_version: "encryptionkey@0"
I20260619 18:46:29.501536 9022 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 18:46:29.503788 9037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:29.504935 9022 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:46:29.505064 9022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-1/wal
uuid: "58edcb95bc9c42a69106373fc359f0f1"
format_stamp: "Formatted at 2026-06-19 18:46:29 on dist-test-slave-mld5"
server_key: "4b4376dc83fac237204a9eaa9201520f"
server_key_iv: "be14279eb4d889cc6d29065c5e4d0b89"
server_key_version: "encryptionkey@0"
I20260619 18:46:29.505167 9022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:29.518368 9022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:29.521426 9022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:29.521615 9022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:29.522150 9022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:46:29.522903 9022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:46:29.522974 9022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:29.523039 9022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:46:29.523082 9022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:29.531940 9022 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.66:42197
I20260619 18:46:29.531975 9150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.66:42197 every 8 connection(s)
I20260619 18:46:29.532742 9022 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
I20260619 18:46:29.535671 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 9022
I20260619 18:46:29.535837 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance
I20260619 18:46:29.536185 22077 external_mini_cluster.cc:1428] Setting key 61695cf6a9d0e81d0a60b480b82b7825
Jun 19 18:46:29 dist-test-slave-mld5 krb5kdc[8781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894789, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.66@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.143.67@KRBTEST.COM; defaulting to no policy
I20260619 18:46:29.546275 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:29.534368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:40747 (local address 127.21.143.126:36101)
0619 18:46:29.534499 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:29.534503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:29.535208 (+ 705us) server_negotiation.cc:407] Connection header received
0619 18:46:29.536962 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:29.536966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:29.537012 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:29.537096 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:29.538532 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.539033 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.540253 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.540464 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.540544 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:29.542802 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:29.542805 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:29.542808 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:29.542839 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:29.544267 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:29.544771 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:29.544775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:29.544776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:29.544851 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:29.545194 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:29.545199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:29.545201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:29.545375 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:29.545483 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:29.545997 (+ 514us) server_negotiation.cc:299] Negotiation successful
0619 18:46:29.546110 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 18:46:29.547107 9153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:29.534651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:36101 (local address 127.21.143.66:40747)
0619 18:46:29.535063 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:29.535103 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:29.536746 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:29.537254 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:29.537261 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:29.537730 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:29.538387 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:29.538409 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.539184 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.539190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:29.540136 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:29.540145 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.540587 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.540589 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:29.540718 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:29.541269 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:29.541284 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:29.542645 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:29.544380 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:29.544385 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:29.544399 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:29.544646 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:29.544942 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:29.544946 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:29.544948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:29.545099 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:29.545480 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:29.545487 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:29.545741 (+ 254us) client_negotiation.cc:769] Sending connection context
0619 18:46:29.545956 (+ 215us) client_negotiation.cc:240] Negotiation successful
0619 18:46:29.546261 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.67@KRBTEST.COM" created.
I20260619 18:46:29.548213 9151 heartbeater.cc:347] Connected to a master server at 127.21.143.126:36101
I20260619 18:46:29.548416 9151 heartbeater.cc:464] Registering TS with master...
I20260619 18:46:29.548856 9151 heartbeater.cc:511] Master 127.21.143.126:36101 requested a full tablet report, sending...
I20260619 18:46:29.549799 8826 ts_manager.cc:194] Registered new tserver with Master: 58edcb95bc9c42a69106373fc359f0f1 (127.21.143.66:42197)
I20260619 18:46:29.550347 8826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.66@KRBTEST.COM'} at 127.21.143.66:40747
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab.
Entry for principal kudu/127.21.143.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab.
I20260619 18:46:29.562117 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.67:0
--local_ip_for_outbound_sockets=127.21.143.67
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:36101
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:34755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:29.671197 9156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:29.671533 9156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:29.671604 9156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:29.675640 9156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:29.675695 9156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:29.675796 9156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.67
I20260619 18:46:29.681113 9156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:34755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.67.keytab
--principal=kudu/127.21.143.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:36101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.67
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:29.682284 9156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:29.683419 9156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:29.693128 9164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:29.693128 9161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:29.693200 9156 server_base.cc:1034] running on GCE node
W20260619 18:46:29.693157 9162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:29.693800 9156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:29.694485 9156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:29.695641 9156 hybrid_clock.cc:648] HybridClock initialized: now 1781894789695624 us; error 38 us; skew 500 ppm
Jun 19 18:46:29 dist-test-slave-mld5 krb5kdc[8781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894789, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:29.698523 9156 init.cc:377] Logged in from keytab as kudu/127.21.143.67@KRBTEST.COM (short username kudu)
I20260619 18:46:29.699498 9156 webserver.cc:460] Webserver started at http://127.21.143.67:40017/ using document root <none> and password file <none>
I20260619 18:46:29.700067 9156 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:29.700162 9156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:29.700404 9156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:29.702427 9156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-2/data/instance:
uuid: "d26efbb17b2e47c7a1a98b38d60be75c"
format_stamp: "Formatted at 2026-06-19 18:46:29 on dist-test-slave-mld5"
server_key: "7d5e59eb0a1cde3421685d7ed2a643bf"
server_key_iv: "987cb5ead9c2e5e3e2b6bcc3f1c3f71e"
server_key_version: "encryptionkey@0"
I20260619 18:46:29.702988 9156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance:
uuid: "d26efbb17b2e47c7a1a98b38d60be75c"
format_stamp: "Formatted at 2026-06-19 18:46:29 on dist-test-slave-mld5"
server_key: "7d5e59eb0a1cde3421685d7ed2a643bf"
server_key_iv: "987cb5ead9c2e5e3e2b6bcc3f1c3f71e"
server_key_version: "encryptionkey@0"
I20260619 18:46:29.706542 9156 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 18:46:29.709060 9171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:29.710265 9156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:46:29.710400 9156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-2/wal
uuid: "d26efbb17b2e47c7a1a98b38d60be75c"
format_stamp: "Formatted at 2026-06-19 18:46:29 on dist-test-slave-mld5"
server_key: "7d5e59eb0a1cde3421685d7ed2a643bf"
server_key_iv: "987cb5ead9c2e5e3e2b6bcc3f1c3f71e"
server_key_version: "encryptionkey@0"
I20260619 18:46:29.710508 9156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:29.721735 9156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:29.724890 9156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:29.725085 9156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:29.725618 9156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:46:29.726399 9156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:46:29.726467 9156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:29.726534 9156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:46:29.726576 9156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:29.735219 9156 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.67:44577
I20260619 18:46:29.735249 9284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.67:44577 every 8 connection(s)
I20260619 18:46:29.736104 9156 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
I20260619 18:46:29.737978 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 9156
I20260619 18:46:29.738116 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance
I20260619 18:46:29.738431 22077 external_mini_cluster.cc:1428] Setting key 577473c12036f41e0b427754f88c6995
Jun 19 18:46:29 dist-test-slave-mld5 krb5kdc[8781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894789, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.67@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:46:29.749369 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:29.737803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:42531 (local address 127.21.143.126:36101)
0619 18:46:29.737961 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:29.737967 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:29.738863 (+ 896us) server_negotiation.cc:407] Connection header received
0619 18:46:29.740313 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:29.740316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:29.740368 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:29.740441 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:29.741605 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.742126 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.743250 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.743568 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.743679 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:29.745975 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:29.745979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:29.745982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:29.746005 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:29.747450 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:29.747948 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:29.747952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:29.747954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:29.748029 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:29.748360 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:29.748363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:29.748365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:29.748532 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:29.748633 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:29.749093 (+ 460us) server_negotiation.cc:299] Negotiation successful
0619 18:46:29.749204 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 18:46:29.750095 9287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:29.738245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:36101 (local address 127.21.143.67:42531)
0619 18:46:29.738709 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:29.738770 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:29.740114 (+ 1344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:29.740668 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:29.740675 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:29.741003 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:29.741455 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:29.741467 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.742245 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.742248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:29.743123 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:29.743131 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.743709 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.743713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:29.743887 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:29.744444 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:29.744460 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:29.745819 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:29.747575 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:29.747581 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:29.747593 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:29.747848 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:29.748120 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:29.748123 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:29.748125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:29.748255 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:29.748641 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:29.748650 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:29.748875 (+ 225us) client_negotiation.cc:769] Sending connection context
0619 18:46:29.749052 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 18:46:29.749370 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":133,"threads_started":1}
I20260619 18:46:29.751266 9285 heartbeater.cc:347] Connected to a master server at 127.21.143.126:36101
I20260619 18:46:29.751605 9285 heartbeater.cc:464] Registering TS with master...
I20260619 18:46:29.752317 9285 heartbeater.cc:511] Master 127.21.143.126:36101 requested a full tablet report, sending...
I20260619 18:46:29.753263 8826 ts_manager.cc:194] Registered new tserver with Master: d26efbb17b2e47c7a1a98b38d60be75c (127.21.143.67:44577)
I20260619 18:46:29.753805 8826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.143.67@KRBTEST.COM'} at 127.21.143.67:42531
I20260619 18:46:29.763752 22077 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:46:29.772626 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:29.765990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42036 (local address 127.21.143.126:36101)
0619 18:46:29.766137 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:29.766142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:29.766327 (+ 185us) server_negotiation.cc:407] Connection header received
0619 18:46:29.766406 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:29.766408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:29.766445 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:29.766516 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:29.767106 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.767664 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.768432 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.768621 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.768726 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:29.770002 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:29.770004 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:29.770006 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:29.770023 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:29.771177 (+ 1154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:29.771591 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:29.771593 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:29.771594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:29.771666 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:29.771886 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:29.771888 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:29.771889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:29.772089 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:29.772188 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:29.772354 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 18:46:29.772441 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 18:46:29.775187 8826 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:46:29.778187 8826 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:46:29.791854 9296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:29.786242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:44577 (local address 127.0.0.1:39716)
0619 18:46:29.786563 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:29.786611 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:29.786731 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:29.787811 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:29.787816 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:29.787839 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:46:29.787989 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:29.787995 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.789241 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.789244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:29.790232 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:29.790241 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.791348 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.791351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:29.791452 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:29.791492 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 18:46:29.791611 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 18:46:29.791715 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":71,"threads_started":1}
I20260619 18:46:29.791882 9298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:29.786364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39716 (local address 127.21.143.67:44577)
0619 18:46:29.787319 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:29.787334 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:29.787357 (+ 23us) server_negotiation.cc:407] Connection header received
0619 18:46:29.787433 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:29.787439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:29.787651 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:29.787833 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:46:29.788096 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.789116 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.790373 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.791203 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.791335 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:29.791488 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:29.791659 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 18:46:29.791750 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":821,"thread_start_us":127,"threads_started":1}
I20260619 18:46:29.792591 9299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:29.787075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:42197 (local address 127.0.0.1:52280)
0619 18:46:29.787278 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:29.787292 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:29.787401 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:29.788291 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:29.788295 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:29.788312 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:46:29.788399 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:29.788405 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.790231 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.790234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:29.791028 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:29.791037 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.792250 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.792253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:29.792331 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:29.792352 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 18:46:29.792413 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 18:46:29.792474 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":2,"thread_start_us":86,"threads_started":1}
I20260619 18:46:29.792634 9300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:29.787533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52280 (local address 127.21.143.66:42197)
0619 18:46:29.787859 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:29.787871 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:29.787892 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:46:29.787961 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:29.787965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:29.788145 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:29.788301 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:46:29.788488 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.789617 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.791166 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.792125 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.792252 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:29.792380 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:29.792433 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 18:46:29.792508 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":138,"threads_started":1}
I20260619 18:46:29.793076 9297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:29.786357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:35889 (local address 127.0.0.1:50244)
0619 18:46:29.786915 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:29.786930 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:29.787016 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:29.788763 (+ 1747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:29.788766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:29.788782 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:46:29.788864 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:29.788887 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.790148 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.790151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:29.791373 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:29.791383 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.792693 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.792696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:29.792776 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:29.792812 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 18:46:29.792879 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 18:46:29.792949 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"mutex_wait_us":47,"thread_start_us":90,"threads_started":1}
I20260619 18:46:29.793953 9301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:29.787937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50244 (local address 127.21.143.65:35889)
0619 18:46:29.788351 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:29.788365 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:29.788385 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:46:29.788457 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:29.788462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:29.788626 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:29.788776 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:46:29.788972 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.790011 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.791505 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.792566 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.793479 (+ 913us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:29.793624 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:29.793672 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 18:46:29.793821 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":126,"threads_started":1}
I20260619 18:46:29.795163 9219 tablet_service.cc:1467] Processing CreateTablet for tablet 502421ab3977422fb1d6fdb0e1238300 (DEFAULT_TABLE table=test-table [id=c24d2df1b9af4365a6798d0ff2f35043]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:46:29.795574 9085 tablet_service.cc:1467] Processing CreateTablet for tablet 502421ab3977422fb1d6fdb0e1238300 (DEFAULT_TABLE table=test-table [id=c24d2df1b9af4365a6798d0ff2f35043]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:46:29.796294 9219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 502421ab3977422fb1d6fdb0e1238300. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:29.796315 9085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 502421ab3977422fb1d6fdb0e1238300. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:29.796494 8951 tablet_service.cc:1467] Processing CreateTablet for tablet 502421ab3977422fb1d6fdb0e1238300 (DEFAULT_TABLE table=test-table [id=c24d2df1b9af4365a6798d0ff2f35043]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:46:29.797402 8951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 502421ab3977422fb1d6fdb0e1238300. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:29.802479 9302 tablet_bootstrap.cc:492] T 502421ab3977422fb1d6fdb0e1238300 P 58edcb95bc9c42a69106373fc359f0f1: Bootstrap starting.
I20260619 18:46:29.803025 9303 tablet_bootstrap.cc:492] T 502421ab3977422fb1d6fdb0e1238300 P d26efbb17b2e47c7a1a98b38d60be75c: Bootstrap starting.
I20260619 18:46:29.804359 9304 tablet_bootstrap.cc:492] T 502421ab3977422fb1d6fdb0e1238300 P 2b502f45c71749cfbf7c7fc260f4dda8: Bootstrap starting.
I20260619 18:46:29.804808 9302 tablet_bootstrap.cc:654] T 502421ab3977422fb1d6fdb0e1238300 P 58edcb95bc9c42a69106373fc359f0f1: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:29.805430 9303 tablet_bootstrap.cc:654] T 502421ab3977422fb1d6fdb0e1238300 P d26efbb17b2e47c7a1a98b38d60be75c: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:29.806154 9304 tablet_bootstrap.cc:654] T 502421ab3977422fb1d6fdb0e1238300 P 2b502f45c71749cfbf7c7fc260f4dda8: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:29.806167 9302 log.cc:824] T 502421ab3977422fb1d6fdb0e1238300 P 58edcb95bc9c42a69106373fc359f0f1: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:29.806602 9303 log.cc:824] T 502421ab3977422fb1d6fdb0e1238300 P d26efbb17b2e47c7a1a98b38d60be75c: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:29.807417 9304 log.cc:824] T 502421ab3977422fb1d6fdb0e1238300 P 2b502f45c71749cfbf7c7fc260f4dda8: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:29.808398 9302 tablet_bootstrap.cc:492] T 502421ab3977422fb1d6fdb0e1238300 P 58edcb95bc9c42a69106373fc359f0f1: No bootstrap required, opened a new log
I20260619 18:46:29.808583 9302 ts_tablet_manager.cc:1397] T 502421ab3977422fb1d6fdb0e1238300 P 58edcb95bc9c42a69106373fc359f0f1: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 18:46:29.808645 9303 tablet_bootstrap.cc:492] T 502421ab3977422fb1d6fdb0e1238300 P d26efbb17b2e47c7a1a98b38d60be75c: No bootstrap required, opened a new log
I20260619 18:46:29.808837 9303 ts_tablet_manager.cc:1397] T 502421ab3977422fb1d6fdb0e1238300 P d26efbb17b2e47c7a1a98b38d60be75c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 18:46:29.809470 9304 tablet_bootstrap.cc:492] T 502421ab3977422fb1d6fdb0e1238300 P 2b502f45c71749cfbf7c7fc260f4dda8: No bootstrap required, opened a new log
I20260619 18:46:29.809672 9304 ts_tablet_manager.cc:1397] T 502421ab3977422fb1d6fdb0e1238300 P 2b502f45c71749cfbf7c7fc260f4dda8: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 18:46:29.812011 9303 raft_consensus.cc:348] T 502421ab3977422fb1d6fdb0e1238300 P d26efbb17b2e47c7a1a98b38d60be75c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26efbb17b2e47c7a1a98b38d60be75c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 44577 } } peers { permanent_uuid: "2b502f45c71749cfbf7c7fc260f4dda8" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 35889 } } peers { permanent_uuid: "58edcb95bc9c42a69106373fc359f0f1" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42197 } }
I20260619 18:46:29.812239 9303 raft_consensus.cc:374] T 502421ab3977422fb1d6fdb0e1238300 P d26efbb17b2e47c7a1a98b38d60be75c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:29.812327 9303 raft_consensus.cc:729] T 502421ab3977422fb1d6fdb0e1238300 P d26efbb17b2e47c7a1a98b38d60be75c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d26efbb17b2e47c7a1a98b38d60be75c, State: Initialized, Role: FOLLOWER
I20260619 18:46:29.812461 9302 raft_consensus.cc:348] T 502421ab3977422fb1d6fdb0e1238300 P 58edcb95bc9c42a69106373fc359f0f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26efbb17b2e47c7a1a98b38d60be75c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 44577 } } peers { permanent_uuid: "2b502f45c71749cfbf7c7fc260f4dda8" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 35889 } } peers { permanent_uuid: "58edcb95bc9c42a69106373fc359f0f1" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42197 } }
I20260619 18:46:29.812704 9302 raft_consensus.cc:374] T 502421ab3977422fb1d6fdb0e1238300 P 58edcb95bc9c42a69106373fc359f0f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:29.812801 9302 raft_consensus.cc:729] T 502421ab3977422fb1d6fdb0e1238300 P 58edcb95bc9c42a69106373fc359f0f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58edcb95bc9c42a69106373fc359f0f1, State: Initialized, Role: FOLLOWER
I20260619 18:46:29.812808 9303 consensus_queue.cc:260] T 502421ab3977422fb1d6fdb0e1238300 P d26efbb17b2e47c7a1a98b38d60be75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26efbb17b2e47c7a1a98b38d60be75c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 44577 } } peers { permanent_uuid: "2b502f45c71749cfbf7c7fc260f4dda8" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 35889 } } peers { permanent_uuid: "58edcb95bc9c42a69106373fc359f0f1" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42197 } }
I20260619 18:46:29.813262 9302 consensus_queue.cc:260] T 502421ab3977422fb1d6fdb0e1238300 P 58edcb95bc9c42a69106373fc359f0f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26efbb17b2e47c7a1a98b38d60be75c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 44577 } } peers { permanent_uuid: "2b502f45c71749cfbf7c7fc260f4dda8" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 35889 } } peers { permanent_uuid: "58edcb95bc9c42a69106373fc359f0f1" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42197 } }
I20260619 18:46:29.813447 9285 heartbeater.cc:503] Master 127.21.143.126:36101 was elected leader, sending a full tablet report...
I20260619 18:46:29.813448 9304 raft_consensus.cc:348] T 502421ab3977422fb1d6fdb0e1238300 P 2b502f45c71749cfbf7c7fc260f4dda8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26efbb17b2e47c7a1a98b38d60be75c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 44577 } } peers { permanent_uuid: "2b502f45c71749cfbf7c7fc260f4dda8" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 35889 } } peers { permanent_uuid: "58edcb95bc9c42a69106373fc359f0f1" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42197 } }
I20260619 18:46:29.813789 9304 raft_consensus.cc:374] T 502421ab3977422fb1d6fdb0e1238300 P 2b502f45c71749cfbf7c7fc260f4dda8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:29.813839 9304 raft_consensus.cc:729] T 502421ab3977422fb1d6fdb0e1238300 P 2b502f45c71749cfbf7c7fc260f4dda8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b502f45c71749cfbf7c7fc260f4dda8, State: Initialized, Role: FOLLOWER
I20260619 18:46:29.814270 9304 consensus_queue.cc:260] T 502421ab3977422fb1d6fdb0e1238300 P 2b502f45c71749cfbf7c7fc260f4dda8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26efbb17b2e47c7a1a98b38d60be75c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 44577 } } peers { permanent_uuid: "2b502f45c71749cfbf7c7fc260f4dda8" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 35889 } } peers { permanent_uuid: "58edcb95bc9c42a69106373fc359f0f1" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42197 } }
I20260619 18:46:29.814378 9302 ts_tablet_manager.cc:1428] T 502421ab3977422fb1d6fdb0e1238300 P 58edcb95bc9c42a69106373fc359f0f1: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260619 18:46:29.814692 9151 heartbeater.cc:503] Master 127.21.143.126:36101 was elected leader, sending a full tablet report...
I20260619 18:46:29.814952 9303 ts_tablet_manager.cc:1428] T 502421ab3977422fb1d6fdb0e1238300 P d26efbb17b2e47c7a1a98b38d60be75c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 18:46:29.816295 9304 ts_tablet_manager.cc:1428] T 502421ab3977422fb1d6fdb0e1238300 P 2b502f45c71749cfbf7c7fc260f4dda8: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260619 18:46:29.816650 9017 heartbeater.cc:503] Master 127.21.143.126:36101 was elected leader, sending a full tablet report...
W20260619 18:46:29.829981 9018 tablet.cc:2369] T 502421ab3977422fb1d6fdb0e1238300 P 2b502f45c71749cfbf7c7fc260f4dda8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:46:29.848450 9310 raft_consensus.cc:482] T 502421ab3977422fb1d6fdb0e1238300 P 2b502f45c71749cfbf7c7fc260f4dda8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:46:29.848672 9310 raft_consensus.cc:504] T 502421ab3977422fb1d6fdb0e1238300 P 2b502f45c71749cfbf7c7fc260f4dda8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26efbb17b2e47c7a1a98b38d60be75c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 44577 } } peers { permanent_uuid: "2b502f45c71749cfbf7c7fc260f4dda8" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 35889 } } peers { permanent_uuid: "58edcb95bc9c42a69106373fc359f0f1" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42197 } }
I20260619 18:46:29.849781 9310 leader_election.cc:288] T 502421ab3977422fb1d6fdb0e1238300 P 2b502f45c71749cfbf7c7fc260f4dda8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d26efbb17b2e47c7a1a98b38d60be75c (127.21.143.67:44577), 58edcb95bc9c42a69106373fc359f0f1 (127.21.143.66:42197)
I20260619 18:46:29.853559 9298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:29.850094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:57521 (local address 127.21.143.67:44577)
0619 18:46:29.850217 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:29.850221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:29.850351 (+ 130us) server_negotiation.cc:407] Connection header received
0619 18:46:29.850464 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:29.850468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:29.850524 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:29.850585 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:46:29.850949 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.851538 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.852602 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.852985 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.853074 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:29.853151 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:29.853340 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 18:46:29.853411 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 18:46:29.853588 9300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:29.850122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:56637 (local address 127.21.143.66:42197)
0619 18:46:29.850251 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:29.850256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:29.850443 (+ 187us) server_negotiation.cc:407] Connection header received
0619 18:46:29.850524 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:29.850526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:29.850583 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:29.850653 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:46:29.851020 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.851543 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.852642 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.853000 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.853141 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:29.853194 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:29.853393 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 18:46:29.853464 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 18:46:29.853602 9311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:29.849978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:44577 (local address 127.21.143.65:57521)
0619 18:46:29.850250 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:29.850264 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:29.850366 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:29.850615 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:29.850620 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:29.850640 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:46:29.850829 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:29.850836 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.851634 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.851638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:29.852485 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:29.852494 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.853108 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.853112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:29.853205 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:29.853235 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 18:46:29.853302 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 18:46:29.853388 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":24,"thread_start_us":77,"threads_started":1}
I20260619 18:46:29.853609 9312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:29.849987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:42197 (local address 127.21.143.65:56637)
0619 18:46:29.850337 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:29.850351 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:29.850451 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:29.850677 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:29.850680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:29.850697 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:46:29.850863 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:29.850868 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.851659 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.851662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:29.852497 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:29.852505 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.853155 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.853160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:29.853249 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:29.853267 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 18:46:29.853356 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 18:46:29.853418 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":70,"thread_start_us":100,"threads_started":1}
I20260619 18:46:29.854327 9239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502421ab3977422fb1d6fdb0e1238300" candidate_uuid: "2b502f45c71749cfbf7c7fc260f4dda8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d26efbb17b2e47c7a1a98b38d60be75c" is_pre_election: true
I20260619 18:46:29.854331 9105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502421ab3977422fb1d6fdb0e1238300" candidate_uuid: "2b502f45c71749cfbf7c7fc260f4dda8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58edcb95bc9c42a69106373fc359f0f1" is_pre_election: true
I20260619 18:46:29.854607 9239 raft_consensus.cc:2454] T 502421ab3977422fb1d6fdb0e1238300 P d26efbb17b2e47c7a1a98b38d60be75c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b502f45c71749cfbf7c7fc260f4dda8 in term 0.
I20260619 18:46:29.854632 9105 raft_consensus.cc:2454] T 502421ab3977422fb1d6fdb0e1238300 P 58edcb95bc9c42a69106373fc359f0f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b502f45c71749cfbf7c7fc260f4dda8 in term 0.
I20260619 18:46:29.855100 8905 leader_election.cc:302] T 502421ab3977422fb1d6fdb0e1238300 P 2b502f45c71749cfbf7c7fc260f4dda8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2b502f45c71749cfbf7c7fc260f4dda8, 58edcb95bc9c42a69106373fc359f0f1, d26efbb17b2e47c7a1a98b38d60be75c; no voters:
I20260619 18:46:29.855365 9310 raft_consensus.cc:2781] T 502421ab3977422fb1d6fdb0e1238300 P 2b502f45c71749cfbf7c7fc260f4dda8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:46:29.855443 9310 raft_consensus.cc:482] T 502421ab3977422fb1d6fdb0e1238300 P 2b502f45c71749cfbf7c7fc260f4dda8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:46:29.855473 9310 raft_consensus.cc:3037] T 502421ab3977422fb1d6fdb0e1238300 P 2b502f45c71749cfbf7c7fc260f4dda8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:29.856499 9310 raft_consensus.cc:504] T 502421ab3977422fb1d6fdb0e1238300 P 2b502f45c71749cfbf7c7fc260f4dda8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26efbb17b2e47c7a1a98b38d60be75c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 44577 } } peers { permanent_uuid: "2b502f45c71749cfbf7c7fc260f4dda8" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 35889 } } peers { permanent_uuid: "58edcb95bc9c42a69106373fc359f0f1" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42197 } }
I20260619 18:46:29.856843 9310 leader_election.cc:288] T 502421ab3977422fb1d6fdb0e1238300 P 2b502f45c71749cfbf7c7fc260f4dda8 [CANDIDATE]: Term 1 election: Requested vote from peers d26efbb17b2e47c7a1a98b38d60be75c (127.21.143.67:44577), 58edcb95bc9c42a69106373fc359f0f1 (127.21.143.66:42197)
I20260619 18:46:29.857095 9239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502421ab3977422fb1d6fdb0e1238300" candidate_uuid: "2b502f45c71749cfbf7c7fc260f4dda8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d26efbb17b2e47c7a1a98b38d60be75c"
I20260619 18:46:29.857219 9239 raft_consensus.cc:3037] T 502421ab3977422fb1d6fdb0e1238300 P d26efbb17b2e47c7a1a98b38d60be75c [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:29.857257 9105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502421ab3977422fb1d6fdb0e1238300" candidate_uuid: "2b502f45c71749cfbf7c7fc260f4dda8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58edcb95bc9c42a69106373fc359f0f1"
I20260619 18:46:29.857383 9105 raft_consensus.cc:3037] T 502421ab3977422fb1d6fdb0e1238300 P 58edcb95bc9c42a69106373fc359f0f1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:29.858215 9239 raft_consensus.cc:2454] T 502421ab3977422fb1d6fdb0e1238300 P d26efbb17b2e47c7a1a98b38d60be75c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b502f45c71749cfbf7c7fc260f4dda8 in term 1.
I20260619 18:46:29.858404 9105 raft_consensus.cc:2454] T 502421ab3977422fb1d6fdb0e1238300 P 58edcb95bc9c42a69106373fc359f0f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b502f45c71749cfbf7c7fc260f4dda8 in term 1.
I20260619 18:46:29.858516 8905 leader_election.cc:302] T 502421ab3977422fb1d6fdb0e1238300 P 2b502f45c71749cfbf7c7fc260f4dda8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b502f45c71749cfbf7c7fc260f4dda8, d26efbb17b2e47c7a1a98b38d60be75c; no voters:
I20260619 18:46:29.858685 9310 raft_consensus.cc:2781] T 502421ab3977422fb1d6fdb0e1238300 P 2b502f45c71749cfbf7c7fc260f4dda8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:29.858883 9310 raft_consensus.cc:686] T 502421ab3977422fb1d6fdb0e1238300 P 2b502f45c71749cfbf7c7fc260f4dda8 [term 1 LEADER]: Becoming Leader. State: Replica: 2b502f45c71749cfbf7c7fc260f4dda8, State: Running, Role: LEADER
I20260619 18:46:29.859196 9310 consensus_queue.cc:237] T 502421ab3977422fb1d6fdb0e1238300 P 2b502f45c71749cfbf7c7fc260f4dda8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26efbb17b2e47c7a1a98b38d60be75c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 44577 } } peers { permanent_uuid: "2b502f45c71749cfbf7c7fc260f4dda8" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 35889 } } peers { permanent_uuid: "58edcb95bc9c42a69106373fc359f0f1" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42197 } }
I20260619 18:46:29.862269 8825 catalog_manager.cc:5495] T 502421ab3977422fb1d6fdb0e1238300 P 2b502f45c71749cfbf7c7fc260f4dda8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b502f45c71749cfbf7c7fc260f4dda8 (127.21.143.65). New cstate: current_term: 1 leader_uuid: "2b502f45c71749cfbf7c7fc260f4dda8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26efbb17b2e47c7a1a98b38d60be75c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 44577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b502f45c71749cfbf7c7fc260f4dda8" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 35889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58edcb95bc9c42a69106373fc359f0f1" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42197 } health_report { overall_health: UNKNOWN } } }
I20260619 18:46:29.888491 9301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:29.885367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50246 (local address 127.21.143.65:35889)
0619 18:46:29.885499 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:29.885503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:29.885516 (+ 13us) server_negotiation.cc:407] Connection header received
0619 18:46:29.885647 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:29.885649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:29.885707 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:29.885797 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:46:29.886002 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.886491 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.887257 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:29.887475 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:29.887549 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:29.887795 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:46:29.888196 (+ 401us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:46:29.888290 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:29.888344 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 18:46:29.888389 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:46:29.894389 9105 raft_consensus.cc:1261] T 502421ab3977422fb1d6fdb0e1238300 P 58edcb95bc9c42a69106373fc359f0f1 [term 1 FOLLOWER]: Refusing update from remote peer 2b502f45c71749cfbf7c7fc260f4dda8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:46:29.894389 9239 raft_consensus.cc:1261] T 502421ab3977422fb1d6fdb0e1238300 P d26efbb17b2e47c7a1a98b38d60be75c [term 1 FOLLOWER]: Refusing update from remote peer 2b502f45c71749cfbf7c7fc260f4dda8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:46:29.895141 9313 consensus_queue.cc:1035] T 502421ab3977422fb1d6fdb0e1238300 P 2b502f45c71749cfbf7c7fc260f4dda8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d26efbb17b2e47c7a1a98b38d60be75c" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 44577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:46:29.895349 9310 consensus_queue.cc:1035] T 502421ab3977422fb1d6fdb0e1238300 P 2b502f45c71749cfbf7c7fc260f4dda8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58edcb95bc9c42a69106373fc359f0f1" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 42197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:46:29.911756 8822 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:46:29.911948 8822 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:46:29.914477 8822 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef059973c2354c0daafefb707aa70d27: Sending DeleteTablet for 3 replicas of tablet 502421ab3977422fb1d6fdb0e1238300
I20260619 18:46:29.915218 9085 tablet_service.cc:1514] Processing DeleteTablet for tablet 502421ab3977422fb1d6fdb0e1238300 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:46:29 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:52280
I20260619 18:46:29.915208 8951 tablet_service.cc:1514] Processing DeleteTablet for tablet 502421ab3977422fb1d6fdb0e1238300 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:46:29 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:50244
I20260619 18:46:29.915719 9326 tablet_replica.cc:330] T 502421ab3977422fb1d6fdb0e1238300 P 2b502f45c71749cfbf7c7fc260f4dda8: stopping tablet replica
I20260619 18:46:29.915721 9219 tablet_service.cc:1514] Processing DeleteTablet for tablet 502421ab3977422fb1d6fdb0e1238300 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:46:29 UTC) from {username='kudu', principal='kudu/127.21.143.126@KRBTEST.COM'} at 127.0.0.1:39716
I20260619 18:46:29.916004 9326 raft_consensus.cc:2229] T 502421ab3977422fb1d6fdb0e1238300 P 2b502f45c71749cfbf7c7fc260f4dda8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:46:29.916158 9328 tablet_replica.cc:330] T 502421ab3977422fb1d6fdb0e1238300 P d26efbb17b2e47c7a1a98b38d60be75c: stopping tablet replica
I20260619 18:46:29.916332 9326 raft_consensus.cc:2258] T 502421ab3977422fb1d6fdb0e1238300 P 2b502f45c71749cfbf7c7fc260f4dda8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:46:29.916430 9328 raft_consensus.cc:2229] T 502421ab3977422fb1d6fdb0e1238300 P d26efbb17b2e47c7a1a98b38d60be75c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:46:29.916673 9327 tablet_replica.cc:330] T 502421ab3977422fb1d6fdb0e1238300 P 58edcb95bc9c42a69106373fc359f0f1: stopping tablet replica
I20260619 18:46:29.916713 9328 raft_consensus.cc:2258] T 502421ab3977422fb1d6fdb0e1238300 P d26efbb17b2e47c7a1a98b38d60be75c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:46:29.916944 9327 raft_consensus.cc:2229] T 502421ab3977422fb1d6fdb0e1238300 P 58edcb95bc9c42a69106373fc359f0f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:46:29.917155 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 8888
I20260619 18:46:29.917213 9327 raft_consensus.cc:2258] T 502421ab3977422fb1d6fdb0e1238300 P 58edcb95bc9c42a69106373fc359f0f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:46:29.918598 9328 ts_tablet_manager.cc:1905] T 502421ab3977422fb1d6fdb0e1238300 P d26efbb17b2e47c7a1a98b38d60be75c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:46:29.918789 9327 ts_tablet_manager.cc:1905] T 502421ab3977422fb1d6fdb0e1238300 P 58edcb95bc9c42a69106373fc359f0f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:46:29.920953 9328 ts_tablet_manager.cc:1918] T 502421ab3977422fb1d6fdb0e1238300 P d26efbb17b2e47c7a1a98b38d60be75c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:46:29.921059 9328 log.cc:1196] T 502421ab3977422fb1d6fdb0e1238300 P d26efbb17b2e47c7a1a98b38d60be75c: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-2/wal/wals/502421ab3977422fb1d6fdb0e1238300
I20260619 18:46:29.921382 9328 ts_tablet_manager.cc:1939] T 502421ab3977422fb1d6fdb0e1238300 P d26efbb17b2e47c7a1a98b38d60be75c: Deleting consensus metadata
I20260619 18:46:29.921834 9327 ts_tablet_manager.cc:1918] T 502421ab3977422fb1d6fdb0e1238300 P 58edcb95bc9c42a69106373fc359f0f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:46:29.921933 9327 log.cc:1196] T 502421ab3977422fb1d6fdb0e1238300 P 58edcb95bc9c42a69106373fc359f0f1: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894542572984-22077-0/minicluster-data/ts-1/wal/wals/502421ab3977422fb1d6fdb0e1238300
I20260619 18:46:29.922248 9327 ts_tablet_manager.cc:1939] T 502421ab3977422fb1d6fdb0e1238300 P 58edcb95bc9c42a69106373fc359f0f1: Deleting consensus metadata
I20260619 18:46:29.922508 8811 catalog_manager.cc:4841] TS d26efbb17b2e47c7a1a98b38d60be75c (127.21.143.67:44577): tablet 502421ab3977422fb1d6fdb0e1238300 (table test-table [id=c24d2df1b9af4365a6798d0ff2f35043]) successfully deleted
I20260619 18:46:29.923507 8812 catalog_manager.cc:4841] TS 58edcb95bc9c42a69106373fc359f0f1 (127.21.143.66:42197): tablet 502421ab3977422fb1d6fdb0e1238300 (table test-table [id=c24d2df1b9af4365a6798d0ff2f35043]) successfully deleted
W20260619 18:46:29.925237 8810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.143.65:35889 (error 108)
I20260619 18:46:29.925319 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 9022
I20260619 18:46:29.927021 9299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:29.926678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:35889 (local address 127.0.0.1:50256)
0619 18:46:29.926846 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:29.926923 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.65:35889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260619 18:46:29.927196 8810 catalog_manager.cc:4584] TS 2b502f45c71749cfbf7c7fc260f4dda8 (127.21.143.65:35889): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 502421ab3977422fb1d6fdb0e1238300: Network error: Client connection negotiation failed: client connection to 127.21.143.65:35889: connect: Connection refused (error 111)
I20260619 18:46:29.931002 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 9156
I20260619 18:46:29.936506 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 8794
2026-06-19T18:46:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2229 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T18:46:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:29Z Disabled control of system clock
I20260619 18:46:29.966538 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:34323
--webserver_interface=127.21.143.126
--webserver_port=0
--builtin_ntp_servers=127.21.143.84:33675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:34323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 18:46:30.074186 9333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:30.074445 9333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:30.074479 9333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:30.078430 9333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:30.078490 9333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:30.078504 9333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:30.078521 9333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:30.078536 9333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:30.083395 9333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:33675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:34323
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:34323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:30.084380 9333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:30.085680 9333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:30.093813 9341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:30.093833 9338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:30.093871 9333 server_base.cc:1034] running on GCE node
W20260619 18:46:30.093833 9339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:30.094542 9333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:30.095606 9333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:30.096760 9333 hybrid_clock.cc:648] HybridClock initialized: now 1781894790096737 us; error 35 us; skew 500 ppm
I20260619 18:46:30.098439 9333 webserver.cc:460] Webserver started at http://127.21.143.126:45109/ using document root <none> and password file <none>
I20260619 18:46:30.099006 9333 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:30.099064 9333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:30.099290 9333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:30.101401 9333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "cf8f536a1ab64dc7b922655941ee1f7a"
format_stamp: "Formatted at 2026-06-19 18:46:30 on dist-test-slave-mld5"
server_key: "5e0c80a87a16df2c96d2b23f08ab5a78"
server_key_iv: "ebec554bd5255f5cf11571050fd62605"
server_key_version: "encryptionkey@0"
I20260619 18:46:30.101981 9333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "cf8f536a1ab64dc7b922655941ee1f7a"
format_stamp: "Formatted at 2026-06-19 18:46:30 on dist-test-slave-mld5"
server_key: "5e0c80a87a16df2c96d2b23f08ab5a78"
server_key_iv: "ebec554bd5255f5cf11571050fd62605"
server_key_version: "encryptionkey@0"
I20260619 18:46:30.105480 9333 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260619 18:46:30.107738 9347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:30.108881 9333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:46:30.109014 9333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "cf8f536a1ab64dc7b922655941ee1f7a"
format_stamp: "Formatted at 2026-06-19 18:46:30 on dist-test-slave-mld5"
server_key: "5e0c80a87a16df2c96d2b23f08ab5a78"
server_key_iv: "ebec554bd5255f5cf11571050fd62605"
server_key_version: "encryptionkey@0"
I20260619 18:46:30.109138 9333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:30.124095 9333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:30.124732 9333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:30.124917 9333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:30.131646 9399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:34323 every 8 connection(s)
I20260619 18:46:30.131656 9333 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:34323
I20260619 18:46:30.132547 9333 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:30.135429 9400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:30.140643 9400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf8f536a1ab64dc7b922655941ee1f7a: Bootstrap starting.
I20260619 18:46:30.141933 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 9333
I20260619 18:46:30.142073 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:46:30.142393 22077 external_mini_cluster.cc:1428] Setting key 7426aa82503cf506bcf8981522817052
I20260619 18:46:30.143230 9400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf8f536a1ab64dc7b922655941ee1f7a: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:30.144065 9400 log.cc:824] T 00000000000000000000000000000000 P cf8f536a1ab64dc7b922655941ee1f7a: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:30.146271 9400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf8f536a1ab64dc7b922655941ee1f7a: No bootstrap required, opened a new log
I20260619 18:46:30.150462 9400 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf8f536a1ab64dc7b922655941ee1f7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8f536a1ab64dc7b922655941ee1f7a" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 34323 } }
I20260619 18:46:30.150736 9400 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf8f536a1ab64dc7b922655941ee1f7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:30.150796 9400 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf8f536a1ab64dc7b922655941ee1f7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf8f536a1ab64dc7b922655941ee1f7a, State: Initialized, Role: FOLLOWER
I20260619 18:46:30.151338 9400 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf8f536a1ab64dc7b922655941ee1f7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8f536a1ab64dc7b922655941ee1f7a" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 34323 } }
I20260619 18:46:30.151497 9400 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf8f536a1ab64dc7b922655941ee1f7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:30.151561 9400 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf8f536a1ab64dc7b922655941ee1f7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:30.151650 9400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf8f536a1ab64dc7b922655941ee1f7a [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:30.152462 9403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:30.143512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55866 (local address 127.21.143.126:34323)
0619 18:46:30.144199 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:30.144218 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:30.144258 (+ 40us) server_negotiation.cc:407] Connection header received
0619 18:46:30.145006 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:30.145027 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:30.145423 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:30.145777 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:30.146203 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:30.147673 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:30.148537 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:30.149246 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:30.149437 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:30.150170 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:30.150175 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:30.150191 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:30.150673 (+ 482us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:30.150747 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:30.150789 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:30.150887 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:30.151056 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 18:46:30.151484 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":106,"threads_started":1}
I20260619 18:46:30.152789 9400 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf8f536a1ab64dc7b922655941ee1f7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8f536a1ab64dc7b922655941ee1f7a" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 34323 } }
I20260619 18:46:30.153119 9400 leader_election.cc:302] T 00000000000000000000000000000000 P cf8f536a1ab64dc7b922655941ee1f7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf8f536a1ab64dc7b922655941ee1f7a; no voters:
I20260619 18:46:30.153407 9400 leader_election.cc:288] T 00000000000000000000000000000000 P cf8f536a1ab64dc7b922655941ee1f7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:30.153554 9405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf8f536a1ab64dc7b922655941ee1f7a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:30.153782 9405 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf8f536a1ab64dc7b922655941ee1f7a [term 1 LEADER]: Becoming Leader. State: Replica: cf8f536a1ab64dc7b922655941ee1f7a, State: Running, Role: LEADER
I20260619 18:46:30.154080 9405 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf8f536a1ab64dc7b922655941ee1f7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8f536a1ab64dc7b922655941ee1f7a" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 34323 } }
I20260619 18:46:30.154667 9400 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf8f536a1ab64dc7b922655941ee1f7a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:30.155835 9406 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf8f536a1ab64dc7b922655941ee1f7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf8f536a1ab64dc7b922655941ee1f7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8f536a1ab64dc7b922655941ee1f7a" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 34323 } } }
I20260619 18:46:30.155988 9406 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf8f536a1ab64dc7b922655941ee1f7a [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:30.155835 9407 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf8f536a1ab64dc7b922655941ee1f7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf8f536a1ab64dc7b922655941ee1f7a. Latest consensus state: current_term: 1 leader_uuid: "cf8f536a1ab64dc7b922655941ee1f7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8f536a1ab64dc7b922655941ee1f7a" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 34323 } } }
I20260619 18:46:30.156080 9407 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf8f536a1ab64dc7b922655941ee1f7a [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:30.156450 9414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:30.159924 9414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:30.164664 9414 catalog_manager.cc:1347] Generated new cluster ID: 2b364fe91a3a4a3e8f1c80d79b2bbfc9
I20260619 18:46:30.164727 9414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:30.225996 9414 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:30.226917 9414 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:30.233180 9414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf8f536a1ab64dc7b922655941ee1f7a: Generated new TSK 0
I20260619 18:46:30.233824 9414 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:30.244390 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.65:0
--local_ip_for_outbound_sockets=127.21.143.65
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:34323
--builtin_ntp_servers=127.21.143.84:33675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 18:46:30.353358 9424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:30.353597 9424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:30.353659 9424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:30.358050 9424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:30.358104 9424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:30.358196 9424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.65
I20260619 18:46:30.362959 9424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:33675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.143.65
--webserver_port=0
--tserver_master_addrs=127.21.143.126:34323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.65
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:30.364074 9424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:30.365449 9424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:30.374889 9429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:30.374920 9430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:30.375072 9424 server_base.cc:1034] running on GCE node
W20260619 18:46:30.374889 9432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:30.375559 9424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:30.376178 9424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:30.377362 9424 hybrid_clock.cc:648] HybridClock initialized: now 1781894790377336 us; error 50 us; skew 500 ppm
I20260619 18:46:30.379148 9424 webserver.cc:460] Webserver started at http://127.21.143.65:45407/ using document root <none> and password file <none>
I20260619 18:46:30.379840 9424 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:30.379945 9424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:30.380158 9424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:30.382211 9424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-0/data/instance:
uuid: "4d2454d28b9b4af29710e399124afac7"
format_stamp: "Formatted at 2026-06-19 18:46:30 on dist-test-slave-mld5"
server_key: "2ebcb7bbc45fe2629e698b1908dffd1f"
server_key_iv: "69d53c93599ef3d9ec981417785b8636"
server_key_version: "encryptionkey@0"
I20260619 18:46:30.382790 9424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance:
uuid: "4d2454d28b9b4af29710e399124afac7"
format_stamp: "Formatted at 2026-06-19 18:46:30 on dist-test-slave-mld5"
server_key: "2ebcb7bbc45fe2629e698b1908dffd1f"
server_key_iv: "69d53c93599ef3d9ec981417785b8636"
server_key_version: "encryptionkey@0"
I20260619 18:46:30.386310 9424 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 18:46:30.388602 9438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:30.389755 9424 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:46:30.389874 9424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-0/wal
uuid: "4d2454d28b9b4af29710e399124afac7"
format_stamp: "Formatted at 2026-06-19 18:46:30 on dist-test-slave-mld5"
server_key: "2ebcb7bbc45fe2629e698b1908dffd1f"
server_key_iv: "69d53c93599ef3d9ec981417785b8636"
server_key_version: "encryptionkey@0"
I20260619 18:46:30.389976 9424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:30.400298 9424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:30.400872 9424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:30.401060 9424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:30.401572 9424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:46:30.402423 9424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:46:30.402493 9424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:30.402556 9424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:46:30.402611 9424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:30.411159 9424 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.65:36603
I20260619 18:46:30.411186 9551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.65:36603 every 8 connection(s)
I20260619 18:46:30.412143 9424 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-0/data/info.pb
I20260619 18:46:30.420274 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 9424
I20260619 18:46:30.420418 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-0/wal/instance
I20260619 18:46:30.420706 22077 external_mini_cluster.cc:1428] Setting key 04969d91ee75c848b443a13322f5d735
I20260619 18:46:30.421578 9403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:30.414020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.65:46457 (local address 127.21.143.126:34323)
0619 18:46:30.414157 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:30.414160 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:30.415325 (+ 1165us) server_negotiation.cc:407] Connection header received
0619 18:46:30.416768 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:30.416771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:30.416832 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:30.416936 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:30.417812 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:30.418361 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:30.419538 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:30.419772 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:30.419884 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:30.421035 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:30.421039 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:30.421043 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:30.421056 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:30.421076 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:30.421087 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:30.421161 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:30.421385 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 18:46:30.421443 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:46:30.422598 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:30.414447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:34323 (local address 127.21.143.65:46457)
0619 18:46:30.415167 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:30.415206 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:30.416550 (+ 1344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:30.417108 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:30.417115 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:30.417199 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:30.417674 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:30.417686 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:30.418487 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:30.418495 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:30.419422 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:30.419431 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:30.419928 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:30.419932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:30.420100 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:30.420742 (+ 642us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:46:30.420744 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:30.420809 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:46:30.420888 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:30.421200 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:30.421209 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:30.421217 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 18:46:30.421359 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 18:46:30.421590 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":117,"threads_started":1}
I20260619 18:46:30.422717 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.66:0
--local_ip_for_outbound_sockets=127.21.143.66
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:34323
--builtin_ntp_servers=127.21.143.84:33675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 18:46:30.423763 9552 heartbeater.cc:347] Connected to a master server at 127.21.143.126:34323
I20260619 18:46:30.423990 9552 heartbeater.cc:464] Registering TS with master...
I20260619 18:46:30.424604 9552 heartbeater.cc:511] Master 127.21.143.126:34323 requested a full tablet report, sending...
I20260619 18:46:30.426033 9364 ts_manager.cc:194] Registered new tserver with Master: 4d2454d28b9b4af29710e399124afac7 (127.21.143.65:36603)
I20260619 18:46:30.427125 9364 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.143.65:46457
W20260619 18:46:30.529827 9555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:30.530059 9555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:30.530097 9555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:30.534003 9555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:30.534044 9555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:30.534129 9555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.66
I20260619 18:46:30.538792 9555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:33675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.143.66
--webserver_port=0
--tserver_master_addrs=127.21.143.126:34323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.66
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:30.539786 9555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:30.541139 9555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:30.550184 9561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:30.550184 9563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:30.550230 9560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:30.550650 9555 server_base.cc:1034] running on GCE node
I20260619 18:46:30.550997 9555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:30.551596 9555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:30.552768 9555 hybrid_clock.cc:648] HybridClock initialized: now 1781894790552739 us; error 42 us; skew 500 ppm
I20260619 18:46:30.554559 9555 webserver.cc:460] Webserver started at http://127.21.143.66:46373/ using document root <none> and password file <none>
I20260619 18:46:30.555109 9555 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:30.555203 9555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:30.555465 9555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:30.557389 9555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-1/data/instance:
uuid: "5997d90e4857497ba50c7a29a2456a1b"
format_stamp: "Formatted at 2026-06-19 18:46:30 on dist-test-slave-mld5"
server_key: "5b5ac28fddd77d52f0d090d97263c4aa"
server_key_iv: "5e1acb754527c480c1a8c3443716b56e"
server_key_version: "encryptionkey@0"
I20260619 18:46:30.557859 9555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance:
uuid: "5997d90e4857497ba50c7a29a2456a1b"
format_stamp: "Formatted at 2026-06-19 18:46:30 on dist-test-slave-mld5"
server_key: "5b5ac28fddd77d52f0d090d97263c4aa"
server_key_iv: "5e1acb754527c480c1a8c3443716b56e"
server_key_version: "encryptionkey@0"
I20260619 18:46:30.562690 9555 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260619 18:46:30.565516 9569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:30.566661 9555 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:46:30.566751 9555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-1/wal
uuid: "5997d90e4857497ba50c7a29a2456a1b"
format_stamp: "Formatted at 2026-06-19 18:46:30 on dist-test-slave-mld5"
server_key: "5b5ac28fddd77d52f0d090d97263c4aa"
server_key_iv: "5e1acb754527c480c1a8c3443716b56e"
server_key_version: "encryptionkey@0"
I20260619 18:46:30.566831 9555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:30.577760 9555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:30.578267 9555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:30.578394 9555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:30.578835 9555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:46:30.579726 9555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:46:30.579771 9555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:30.579806 9555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:46:30.579828 9555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:30.588178 9555 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.66:35915
I20260619 18:46:30.588232 9682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.66:35915 every 8 connection(s)
I20260619 18:46:30.589003 9555 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-1/data/info.pb
I20260619 18:46:30.598762 9403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:30.590641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:36637 (local address 127.21.143.126:34323)
0619 18:46:30.590780 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:30.590784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:30.591768 (+ 984us) server_negotiation.cc:407] Connection header received
0619 18:46:30.593571 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:30.593576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:30.593635 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:30.593735 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:30.594764 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:30.595707 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:30.596818 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:30.597054 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:30.597166 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:30.598255 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:30.598259 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:30.598262 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:30.598275 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:30.598295 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:30.598311 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:30.598391 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:30.598549 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 18:46:30.598609 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 18:46:30.599165 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 9555
I20260619 18:46:30.599265 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-1/wal/instance
I20260619 18:46:30.599591 22077 external_mini_cluster.cc:1428] Setting key 7170e8a5f7fd5778dafabaf35849ee80
I20260619 18:46:30.599649 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:30.591001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:34323 (local address 127.21.143.66:36637)
0619 18:46:30.591638 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:30.591674 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:30.593286 (+ 1612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:30.593898 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:30.593904 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:30.593983 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:30.594611 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:30.594632 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:30.595829 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:30.595832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:30.596689 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:30.596699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:30.597201 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:30.597205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:30.597378 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:30.597998 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:46:30.597999 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:30.598037 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:46:30.598096 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:30.598392 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:30.598403 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:30.598412 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 18:46:30.598545 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 18:46:30.598802 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":112,"threads_started":1}
I20260619 18:46:30.600826 9683 heartbeater.cc:347] Connected to a master server at 127.21.143.126:34323
I20260619 18:46:30.601186 9683 heartbeater.cc:464] Registering TS with master...
I20260619 18:46:30.601794 9683 heartbeater.cc:511] Master 127.21.143.126:34323 requested a full tablet report, sending...
I20260619 18:46:30.601940 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.143.67:0
--local_ip_for_outbound_sockets=127.21.143.67
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:34323
--builtin_ntp_servers=127.21.143.84:33675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 18:46:30.602905 9363 ts_manager.cc:194] Registered new tserver with Master: 5997d90e4857497ba50c7a29a2456a1b (127.21.143.66:35915)
I20260619 18:46:30.603698 9363 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.143.66:36637
W20260619 18:46:30.709926 9686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:30.710175 9686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:30.710230 9686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:30.714262 9686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:30.714313 9686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:30.714365 9686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.143.67
I20260619 18:46:30.719192 9686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:33675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.143.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.143.67
--webserver_port=0
--tserver_master_addrs=127.21.143.126:34323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.143.67
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:30.720171 9686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:30.721436 9686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:30.730640 9692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:30.730625 9691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:30.730695 9686 server_base.cc:1034] running on GCE node
W20260619 18:46:30.730659 9694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:30.731377 9686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:30.731982 9686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:30.733150 9686 hybrid_clock.cc:648] HybridClock initialized: now 1781894790733114 us; error 48 us; skew 500 ppm
I20260619 18:46:30.735486 9686 webserver.cc:460] Webserver started at http://127.21.143.67:43917/ using document root <none> and password file <none>
I20260619 18:46:30.736248 9686 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:30.736359 9686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:30.736618 9686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:30.739574 9686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-2/data/instance:
uuid: "ea46bc7bc8ad489b90e8bcb3eba7b3ad"
format_stamp: "Formatted at 2026-06-19 18:46:30 on dist-test-slave-mld5"
server_key: "7fdf8c6a967e7f119b11cb8df5fe9201"
server_key_iv: "3ee60b6215b7b310c45f30b74a85937f"
server_key_version: "encryptionkey@0"
I20260619 18:46:30.740234 9686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance:
uuid: "ea46bc7bc8ad489b90e8bcb3eba7b3ad"
format_stamp: "Formatted at 2026-06-19 18:46:30 on dist-test-slave-mld5"
server_key: "7fdf8c6a967e7f119b11cb8df5fe9201"
server_key_iv: "3ee60b6215b7b310c45f30b74a85937f"
server_key_version: "encryptionkey@0"
I20260619 18:46:30.743744 9686 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:46:30.746004 9700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:30.747284 9686 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:46:30.747429 9686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-2/wal
uuid: "ea46bc7bc8ad489b90e8bcb3eba7b3ad"
format_stamp: "Formatted at 2026-06-19 18:46:30 on dist-test-slave-mld5"
server_key: "7fdf8c6a967e7f119b11cb8df5fe9201"
server_key_iv: "3ee60b6215b7b310c45f30b74a85937f"
server_key_version: "encryptionkey@0"
I20260619 18:46:30.747555 9686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:30.773958 9686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:30.774919 9686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:30.775127 9686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:30.775969 9686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:46:30.777436 9686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:46:30.777503 9686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:30.777563 9686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:46:30.777606 9686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:30.787160 9686 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.67:41457
I20260619 18:46:30.787181 9813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.67:41457 every 8 connection(s)
I20260619 18:46:30.788076 9686 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-2/data/info.pb
I20260619 18:46:30.797079 9403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:30.789878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.67:59547 (local address 127.21.143.126:34323)
0619 18:46:30.790044 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:30.790049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:30.791064 (+ 1015us) server_negotiation.cc:407] Connection header received
0619 18:46:30.792496 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:30.792498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:30.792546 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:30.792620 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:30.793604 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:30.794223 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:30.795358 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:30.795573 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:30.795661 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:30.796680 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:30.796683 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:30.796685 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:30.796697 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:30.796716 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:30.796724 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:30.796779 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:30.796912 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 18:46:30.796963 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 18:46:30.797932 9816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:30.790303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:34323 (local address 127.21.143.67:59547)
0619 18:46:30.790917 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:30.790958 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:30.792308 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:30.792849 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:30.792864 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:30.792952 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:30.793473 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:30.793486 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:30.794327 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:30.794329 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:30.795216 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:30.795224 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:30.795707 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:30.795710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:30.795849 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:30.796428 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:46:30.796429 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:30.796469 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:46:30.796532 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:30.796794 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:30.796800 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:30.796803 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 18:46:30.796897 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 18:46:30.797113 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":106,"threads_started":1}
I20260619 18:46:30.798386 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 9686
I20260619 18:46:30.798507 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-2/wal/instance
I20260619 18:46:30.798817 22077 external_mini_cluster.cc:1428] Setting key 55f5a640bc54553bb13be1a7dfd4b82b
I20260619 18:46:30.799140 9814 heartbeater.cc:347] Connected to a master server at 127.21.143.126:34323
I20260619 18:46:30.799435 9814 heartbeater.cc:464] Registering TS with master...
I20260619 18:46:30.800047 9814 heartbeater.cc:511] Master 127.21.143.126:34323 requested a full tablet report, sending...
I20260619 18:46:30.801084 9363 ts_manager.cc:194] Registered new tserver with Master: ea46bc7bc8ad489b90e8bcb3eba7b3ad (127.21.143.67:41457)
I20260619 18:46:30.801656 9363 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.143.67:59547
I20260619 18:46:30.812534 22077 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:46:30.817971 9403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:30.814519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55882 (local address 127.21.143.126:34323)
0619 18:46:30.814666 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:30.814671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:30.814824 (+ 153us) server_negotiation.cc:407] Connection header received
0619 18:46:30.814916 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:30.814918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:30.814957 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:30.815021 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:30.815238 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:30.815841 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:30.816599 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:30.816807 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:30.816879 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:30.817544 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:30.817547 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:30.817549 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:30.817559 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:30.817576 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:30.817584 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:30.817651 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:30.817731 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 18:46:30.817780 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 18:46:30.820515 9363 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:46:30.823450 9363 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:46:30.837785 9825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:30.831582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:36603 (local address 127.0.0.1:47866)
0619 18:46:30.832410 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:30.832445 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:30.832564 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:30.833367 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:30.833373 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:30.833406 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:46:30.833617 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:30.833623 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:30.835247 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:30.835253 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:30.836289 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:30.836300 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:30.837173 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:30.837176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:30.837382 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:30.837411 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 18:46:30.837499 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 18:46:30.837627 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":246,"spinlock_wait_cycles":9728,"thread_start_us":90,"threads_started":1}
I20260619 18:46:30.837842 9827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:30.832023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37800 (local address 127.21.143.67:41457)
0619 18:46:30.832877 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:30.832885 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:30.832905 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:46:30.832989 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:30.832993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:30.833191 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:30.833378 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:46:30.833729 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:30.834630 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:30.836301 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:30.837004 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:30.837114 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:30.837314 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:30.837600 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 18:46:30.837684 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":775,"thread_start_us":156,"threads_started":1}
I20260619 18:46:30.837842 9829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:30.832560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47866 (local address 127.21.143.65:36603)
0619 18:46:30.832873 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:30.832885 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:30.832907 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:46:30.832983 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:30.832988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:30.833205 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:30.833759 (+ 554us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:46:30.833905 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:30.835095 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:30.836407 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:30.837030 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:30.837371 (+ 341us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:30.837545 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:30.837594 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 18:46:30.837684 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":131,"threads_started":1}
I20260619 18:46:30.837962 9828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:30.832303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:35915 (local address 127.0.0.1:57270)
0619 18:46:30.833077 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:30.833089 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:30.833164 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:30.833839 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:30.833842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:30.833860 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:46:30.833944 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:30.833949 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:30.835279 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:30.835282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:30.836550 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:30.836559 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:30.837554 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:30.837560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:30.837664 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:30.837684 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 18:46:30.837732 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 18:46:30.837782 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":76,"thread_start_us":75,"threads_started":1}
I20260619 18:46:30.838742 9830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:30.833059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57270 (local address 127.21.143.66:35915)
0619 18:46:30.833390 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:30.833405 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:30.833426 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:46:30.833495 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:30.833499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:30.833714 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:30.833844 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:46:30.834057 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:30.835166 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:30.836684 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:30.837424 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:30.838095 (+ 671us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:30.838259 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:30.838311 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 18:46:30.838582 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":108,"threads_started":1}
I20260619 18:46:30.838865 9826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:30.831821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:41457 (local address 127.0.0.1:37800)
0619 18:46:30.832384 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:30.832434 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:30.832563 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:30.833369 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:30.833374 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:30.833404 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:46:30.833612 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:30.833619 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:30.834761 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:30.834764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:30.836174 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:30.836185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:30.837135 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:30.837141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:30.837358 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:30.837406 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 18:46:30.837509 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 18:46:30.838718 (+ 1209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":1191,"spinlock_wait_cycles":10624,"thread_start_us":110,"threads_started":1}
I20260619 18:46:30.840921 9617 tablet_service.cc:1467] Processing CreateTablet for tablet 37b1558706024e43a18ca0c888c55558 (DEFAULT_TABLE table=test-table [id=a48959ff1e284c2798394f7d62275698]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:46:30.840920 9486 tablet_service.cc:1467] Processing CreateTablet for tablet 37b1558706024e43a18ca0c888c55558 (DEFAULT_TABLE table=test-table [id=a48959ff1e284c2798394f7d62275698]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:46:30.841825 9486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37b1558706024e43a18ca0c888c55558. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:30.841841 9617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37b1558706024e43a18ca0c888c55558. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:30.843706 9748 tablet_service.cc:1467] Processing CreateTablet for tablet 37b1558706024e43a18ca0c888c55558 (DEFAULT_TABLE table=test-table [id=a48959ff1e284c2798394f7d62275698]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:46:30.844602 9748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37b1558706024e43a18ca0c888c55558. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:30.848927 9831 tablet_bootstrap.cc:492] T 37b1558706024e43a18ca0c888c55558 P 4d2454d28b9b4af29710e399124afac7: Bootstrap starting.
I20260619 18:46:30.849615 9832 tablet_bootstrap.cc:492] T 37b1558706024e43a18ca0c888c55558 P 5997d90e4857497ba50c7a29a2456a1b: Bootstrap starting.
I20260619 18:46:30.851176 9833 tablet_bootstrap.cc:492] T 37b1558706024e43a18ca0c888c55558 P ea46bc7bc8ad489b90e8bcb3eba7b3ad: Bootstrap starting.
I20260619 18:46:30.851351 9831 tablet_bootstrap.cc:654] T 37b1558706024e43a18ca0c888c55558 P 4d2454d28b9b4af29710e399124afac7: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:30.851643 9832 tablet_bootstrap.cc:654] T 37b1558706024e43a18ca0c888c55558 P 5997d90e4857497ba50c7a29a2456a1b: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:30.852344 9832 log.cc:824] T 37b1558706024e43a18ca0c888c55558 P 5997d90e4857497ba50c7a29a2456a1b: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:30.852331 9831 log.cc:824] T 37b1558706024e43a18ca0c888c55558 P 4d2454d28b9b4af29710e399124afac7: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:30.853087 9833 tablet_bootstrap.cc:654] T 37b1558706024e43a18ca0c888c55558 P ea46bc7bc8ad489b90e8bcb3eba7b3ad: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:30.853821 9833 log.cc:824] T 37b1558706024e43a18ca0c888c55558 P ea46bc7bc8ad489b90e8bcb3eba7b3ad: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:30.854239 9832 tablet_bootstrap.cc:492] T 37b1558706024e43a18ca0c888c55558 P 5997d90e4857497ba50c7a29a2456a1b: No bootstrap required, opened a new log
I20260619 18:46:30.854434 9832 ts_tablet_manager.cc:1397] T 37b1558706024e43a18ca0c888c55558 P 5997d90e4857497ba50c7a29a2456a1b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 18:46:30.854799 9831 tablet_bootstrap.cc:492] T 37b1558706024e43a18ca0c888c55558 P 4d2454d28b9b4af29710e399124afac7: No bootstrap required, opened a new log
I20260619 18:46:30.854993 9831 ts_tablet_manager.cc:1397] T 37b1558706024e43a18ca0c888c55558 P 4d2454d28b9b4af29710e399124afac7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 18:46:30.855743 9833 tablet_bootstrap.cc:492] T 37b1558706024e43a18ca0c888c55558 P ea46bc7bc8ad489b90e8bcb3eba7b3ad: No bootstrap required, opened a new log
I20260619 18:46:30.855984 9833 ts_tablet_manager.cc:1397] T 37b1558706024e43a18ca0c888c55558 P ea46bc7bc8ad489b90e8bcb3eba7b3ad: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 18:46:30.857796 9832 raft_consensus.cc:348] T 37b1558706024e43a18ca0c888c55558 P 5997d90e4857497ba50c7a29a2456a1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d2454d28b9b4af29710e399124afac7" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36603 } } peers { permanent_uuid: "ea46bc7bc8ad489b90e8bcb3eba7b3ad" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 41457 } } peers { permanent_uuid: "5997d90e4857497ba50c7a29a2456a1b" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35915 } }
I20260619 18:46:30.858031 9832 raft_consensus.cc:374] T 37b1558706024e43a18ca0c888c55558 P 5997d90e4857497ba50c7a29a2456a1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:30.858100 9832 raft_consensus.cc:729] T 37b1558706024e43a18ca0c888c55558 P 5997d90e4857497ba50c7a29a2456a1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5997d90e4857497ba50c7a29a2456a1b, State: Initialized, Role: FOLLOWER
I20260619 18:46:30.858546 9832 consensus_queue.cc:260] T 37b1558706024e43a18ca0c888c55558 P 5997d90e4857497ba50c7a29a2456a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d2454d28b9b4af29710e399124afac7" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36603 } } peers { permanent_uuid: "ea46bc7bc8ad489b90e8bcb3eba7b3ad" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 41457 } } peers { permanent_uuid: "5997d90e4857497ba50c7a29a2456a1b" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35915 } }
I20260619 18:46:30.859174 9683 heartbeater.cc:503] Master 127.21.143.126:34323 was elected leader, sending a full tablet report...
I20260619 18:46:30.859282 9831 raft_consensus.cc:348] T 37b1558706024e43a18ca0c888c55558 P 4d2454d28b9b4af29710e399124afac7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d2454d28b9b4af29710e399124afac7" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36603 } } peers { permanent_uuid: "ea46bc7bc8ad489b90e8bcb3eba7b3ad" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 41457 } } peers { permanent_uuid: "5997d90e4857497ba50c7a29a2456a1b" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35915 } }
I20260619 18:46:30.859547 9831 raft_consensus.cc:374] T 37b1558706024e43a18ca0c888c55558 P 4d2454d28b9b4af29710e399124afac7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:30.859570 9832 ts_tablet_manager.cc:1428] T 37b1558706024e43a18ca0c888c55558 P 5997d90e4857497ba50c7a29a2456a1b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 18:46:30.859593 9831 raft_consensus.cc:729] T 37b1558706024e43a18ca0c888c55558 P 4d2454d28b9b4af29710e399124afac7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d2454d28b9b4af29710e399124afac7, State: Initialized, Role: FOLLOWER
I20260619 18:46:30.859975 9831 consensus_queue.cc:260] T 37b1558706024e43a18ca0c888c55558 P 4d2454d28b9b4af29710e399124afac7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d2454d28b9b4af29710e399124afac7" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36603 } } peers { permanent_uuid: "ea46bc7bc8ad489b90e8bcb3eba7b3ad" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 41457 } } peers { permanent_uuid: "5997d90e4857497ba50c7a29a2456a1b" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35915 } }
I20260619 18:46:30.860013 9833 raft_consensus.cc:348] T 37b1558706024e43a18ca0c888c55558 P ea46bc7bc8ad489b90e8bcb3eba7b3ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d2454d28b9b4af29710e399124afac7" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36603 } } peers { permanent_uuid: "ea46bc7bc8ad489b90e8bcb3eba7b3ad" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 41457 } } peers { permanent_uuid: "5997d90e4857497ba50c7a29a2456a1b" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35915 } }
I20260619 18:46:30.860255 9833 raft_consensus.cc:374] T 37b1558706024e43a18ca0c888c55558 P ea46bc7bc8ad489b90e8bcb3eba7b3ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:30.860313 9833 raft_consensus.cc:729] T 37b1558706024e43a18ca0c888c55558 P ea46bc7bc8ad489b90e8bcb3eba7b3ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea46bc7bc8ad489b90e8bcb3eba7b3ad, State: Initialized, Role: FOLLOWER
I20260619 18:46:30.860713 9833 consensus_queue.cc:260] T 37b1558706024e43a18ca0c888c55558 P ea46bc7bc8ad489b90e8bcb3eba7b3ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d2454d28b9b4af29710e399124afac7" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36603 } } peers { permanent_uuid: "ea46bc7bc8ad489b90e8bcb3eba7b3ad" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 41457 } } peers { permanent_uuid: "5997d90e4857497ba50c7a29a2456a1b" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35915 } }
I20260619 18:46:30.860879 9831 ts_tablet_manager.cc:1428] T 37b1558706024e43a18ca0c888c55558 P 4d2454d28b9b4af29710e399124afac7: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260619 18:46:30.861464 9833 ts_tablet_manager.cc:1428] T 37b1558706024e43a18ca0c888c55558 P ea46bc7bc8ad489b90e8bcb3eba7b3ad: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 18:46:30.861744 9814 heartbeater.cc:503] Master 127.21.143.126:34323 was elected leader, sending a full tablet report...
I20260619 18:46:30.862211 9552 heartbeater.cc:503] Master 127.21.143.126:34323 was elected leader, sending a full tablet report...
W20260619 18:46:30.913341 9553 tablet.cc:2369] T 37b1558706024e43a18ca0c888c55558 P 4d2454d28b9b4af29710e399124afac7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:46:31.039098 9815 tablet.cc:2369] T 37b1558706024e43a18ca0c888c55558 P ea46bc7bc8ad489b90e8bcb3eba7b3ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:46:31.090060 9684 tablet.cc:2369] T 37b1558706024e43a18ca0c888c55558 P 5997d90e4857497ba50c7a29a2456a1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:46:31.277035 9837 raft_consensus.cc:482] T 37b1558706024e43a18ca0c888c55558 P 5997d90e4857497ba50c7a29a2456a1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:46:31.277252 9837 raft_consensus.cc:504] T 37b1558706024e43a18ca0c888c55558 P 5997d90e4857497ba50c7a29a2456a1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d2454d28b9b4af29710e399124afac7" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36603 } } peers { permanent_uuid: "ea46bc7bc8ad489b90e8bcb3eba7b3ad" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 41457 } } peers { permanent_uuid: "5997d90e4857497ba50c7a29a2456a1b" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35915 } }
I20260619 18:46:31.278577 9837 leader_election.cc:288] T 37b1558706024e43a18ca0c888c55558 P 5997d90e4857497ba50c7a29a2456a1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d2454d28b9b4af29710e399124afac7 (127.21.143.65:36603), ea46bc7bc8ad489b90e8bcb3eba7b3ad (127.21.143.67:41457)
I20260619 18:46:31.282507 9840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:31.278790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.67:41457 (local address 127.21.143.66:56297)
0619 18:46:31.279096 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:31.279112 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:31.279221 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:31.279510 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:31.279513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:31.279539 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:46:31.279676 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:31.279685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:31.280451 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:31.280457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:31.281398 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:31.281407 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:31.282067 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:31.282071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:31.282150 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:31.282178 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 18:46:31.282246 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 18:46:31.282304 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":33,"thread_start_us":86,"threads_started":1}
I20260619 18:46:31.282639 9829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:31.278910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:38485 (local address 127.21.143.65:36603)
0619 18:46:31.279137 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:31.279144 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:31.279341 (+ 197us) server_negotiation.cc:407] Connection header received
0619 18:46:31.279449 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:31.279472 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:31.279529 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:31.279609 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:46:31.279897 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:31.280402 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:31.281580 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:31.281957 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:31.282095 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:31.282143 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:31.282416 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 18:46:31.282503 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260619 18:46:31.282621 9827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:31.278989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.143.66:56297 (local address 127.21.143.67:41457)
0619 18:46:31.279120 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:31.279125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:31.279181 (+ 56us) server_negotiation.cc:407] Connection header received
0619 18:46:31.279345 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:31.279348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:31.279408 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:31.279479 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:46:31.279805 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:31.280338 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:31.281525 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:31.281938 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:31.282055 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:31.282119 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:31.282284 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 18:46:31.282374 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 18:46:31.282827 9841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:31.278789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:36603 (local address 127.21.143.66:38485)
0619 18:46:31.279226 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:31.279239 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:31.279352 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:31.279639 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:31.279641 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:31.279655 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:46:31.279788 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:31.279794 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:31.280521 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:31.280524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:31.281428 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:31.281437 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:31.282128 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:31.282135 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:31.282264 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:31.282286 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 18:46:31.282359 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 18:46:31.282446 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":127,"thread_start_us":79,"threads_started":1}
I20260619 18:46:31.283479 9506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37b1558706024e43a18ca0c888c55558" candidate_uuid: "5997d90e4857497ba50c7a29a2456a1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d2454d28b9b4af29710e399124afac7" is_pre_election: true
I20260619 18:46:31.283483 9768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37b1558706024e43a18ca0c888c55558" candidate_uuid: "5997d90e4857497ba50c7a29a2456a1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea46bc7bc8ad489b90e8bcb3eba7b3ad" is_pre_election: true
I20260619 18:46:31.283807 9768 raft_consensus.cc:2454] T 37b1558706024e43a18ca0c888c55558 P ea46bc7bc8ad489b90e8bcb3eba7b3ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5997d90e4857497ba50c7a29a2456a1b in term 0.
I20260619 18:46:31.283808 9506 raft_consensus.cc:2454] T 37b1558706024e43a18ca0c888c55558 P 4d2454d28b9b4af29710e399124afac7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5997d90e4857497ba50c7a29a2456a1b in term 0.
I20260619 18:46:31.284322 9570 leader_election.cc:302] T 37b1558706024e43a18ca0c888c55558 P 5997d90e4857497ba50c7a29a2456a1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5997d90e4857497ba50c7a29a2456a1b, ea46bc7bc8ad489b90e8bcb3eba7b3ad; no voters:
I20260619 18:46:31.284618 9837 raft_consensus.cc:2781] T 37b1558706024e43a18ca0c888c55558 P 5997d90e4857497ba50c7a29a2456a1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:46:31.284708 9837 raft_consensus.cc:482] T 37b1558706024e43a18ca0c888c55558 P 5997d90e4857497ba50c7a29a2456a1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:46:31.284739 9837 raft_consensus.cc:3037] T 37b1558706024e43a18ca0c888c55558 P 5997d90e4857497ba50c7a29a2456a1b [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:31.285758 9837 raft_consensus.cc:504] T 37b1558706024e43a18ca0c888c55558 P 5997d90e4857497ba50c7a29a2456a1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d2454d28b9b4af29710e399124afac7" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36603 } } peers { permanent_uuid: "ea46bc7bc8ad489b90e8bcb3eba7b3ad" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 41457 } } peers { permanent_uuid: "5997d90e4857497ba50c7a29a2456a1b" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35915 } }
I20260619 18:46:31.286105 9837 leader_election.cc:288] T 37b1558706024e43a18ca0c888c55558 P 5997d90e4857497ba50c7a29a2456a1b [CANDIDATE]: Term 1 election: Requested vote from peers 4d2454d28b9b4af29710e399124afac7 (127.21.143.65:36603), ea46bc7bc8ad489b90e8bcb3eba7b3ad (127.21.143.67:41457)
I20260619 18:46:31.286451 9506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37b1558706024e43a18ca0c888c55558" candidate_uuid: "5997d90e4857497ba50c7a29a2456a1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d2454d28b9b4af29710e399124afac7"
I20260619 18:46:31.286553 9506 raft_consensus.cc:3037] T 37b1558706024e43a18ca0c888c55558 P 4d2454d28b9b4af29710e399124afac7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:31.287485 9768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37b1558706024e43a18ca0c888c55558" candidate_uuid: "5997d90e4857497ba50c7a29a2456a1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea46bc7bc8ad489b90e8bcb3eba7b3ad"
I20260619 18:46:31.287526 9506 raft_consensus.cc:2454] T 37b1558706024e43a18ca0c888c55558 P 4d2454d28b9b4af29710e399124afac7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5997d90e4857497ba50c7a29a2456a1b in term 1.
I20260619 18:46:31.287626 9768 raft_consensus.cc:3037] T 37b1558706024e43a18ca0c888c55558 P ea46bc7bc8ad489b90e8bcb3eba7b3ad [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:31.287930 9573 leader_election.cc:302] T 37b1558706024e43a18ca0c888c55558 P 5997d90e4857497ba50c7a29a2456a1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d2454d28b9b4af29710e399124afac7, 5997d90e4857497ba50c7a29a2456a1b; no voters:
I20260619 18:46:31.288130 9837 raft_consensus.cc:2781] T 37b1558706024e43a18ca0c888c55558 P 5997d90e4857497ba50c7a29a2456a1b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:31.288335 9837 raft_consensus.cc:686] T 37b1558706024e43a18ca0c888c55558 P 5997d90e4857497ba50c7a29a2456a1b [term 1 LEADER]: Becoming Leader. State: Replica: 5997d90e4857497ba50c7a29a2456a1b, State: Running, Role: LEADER
I20260619 18:46:31.288704 9837 consensus_queue.cc:237] T 37b1558706024e43a18ca0c888c55558 P 5997d90e4857497ba50c7a29a2456a1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d2454d28b9b4af29710e399124afac7" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36603 } } peers { permanent_uuid: "ea46bc7bc8ad489b90e8bcb3eba7b3ad" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 41457 } } peers { permanent_uuid: "5997d90e4857497ba50c7a29a2456a1b" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35915 } }
I20260619 18:46:31.288985 9768 raft_consensus.cc:2454] T 37b1558706024e43a18ca0c888c55558 P ea46bc7bc8ad489b90e8bcb3eba7b3ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5997d90e4857497ba50c7a29a2456a1b in term 1.
I20260619 18:46:31.292068 9362 catalog_manager.cc:5495] T 37b1558706024e43a18ca0c888c55558 P 5997d90e4857497ba50c7a29a2456a1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5997d90e4857497ba50c7a29a2456a1b (127.21.143.66). New cstate: current_term: 1 leader_uuid: "5997d90e4857497ba50c7a29a2456a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d2454d28b9b4af29710e399124afac7" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea46bc7bc8ad489b90e8bcb3eba7b3ad" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 41457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5997d90e4857497ba50c7a29a2456a1b" member_type: VOTER last_known_addr { host: "127.21.143.66" port: 35915 } health_report { overall_health: HEALTHY } } }
I20260619 18:46:31.396471 9846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:31.392664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57284 (local address 127.21.143.66:35915)
0619 18:46:31.392903 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:31.392908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:31.392921 (+ 13us) server_negotiation.cc:407] Connection header received
0619 18:46:31.393069 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:31.393071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:31.393126 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:31.393253 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:46:31.393518 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:31.394045 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:31.394942 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:31.395181 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:31.395291 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:31.395655 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:46:31.396081 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:46:31.396218 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:31.396258 (+ 40us) server_negotiation.cc:299] Negotiation successful
0619 18:46:31.396310 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":72,"threads_started":1}
I20260619 18:46:31.402065 9768 raft_consensus.cc:1261] T 37b1558706024e43a18ca0c888c55558 P ea46bc7bc8ad489b90e8bcb3eba7b3ad [term 1 FOLLOWER]: Refusing update from remote peer 5997d90e4857497ba50c7a29a2456a1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:46:31.402069 9506 raft_consensus.cc:1261] T 37b1558706024e43a18ca0c888c55558 P 4d2454d28b9b4af29710e399124afac7 [term 1 FOLLOWER]: Refusing update from remote peer 5997d90e4857497ba50c7a29a2456a1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:46:31.402784 9842 consensus_queue.cc:1035] T 37b1558706024e43a18ca0c888c55558 P 5997d90e4857497ba50c7a29a2456a1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d2454d28b9b4af29710e399124afac7" member_type: VOTER last_known_addr { host: "127.21.143.65" port: 36603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:46:31.402961 9837 consensus_queue.cc:1035] T 37b1558706024e43a18ca0c888c55558 P 5997d90e4857497ba50c7a29a2456a1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea46bc7bc8ad489b90e8bcb3eba7b3ad" member_type: VOTER last_known_addr { host: "127.21.143.67" port: 41457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:46:31.415735 9851 mvcc.cc:205] Tried to move back new op lower bound from 7298641065575976960 to 7298641065121705984. Current Snapshot: MvccSnapshot[applied={T|T < 7298641065575976960 or (T in {7298641065575976960})}]
I20260619 18:46:31.422175 9364 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:46:31.422369 9364 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:46:31.425139 9364 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf8f536a1ab64dc7b922655941ee1f7a: Sending DeleteTablet for 3 replicas of tablet 37b1558706024e43a18ca0c888c55558
I20260619 18:46:31.425900 9486 tablet_service.cc:1514] Processing DeleteTablet for tablet 37b1558706024e43a18ca0c888c55558 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:46:31 UTC) from {username='slave'} at 127.0.0.1:47866
I20260619 18:46:31.426093 9748 tablet_service.cc:1514] Processing DeleteTablet for tablet 37b1558706024e43a18ca0c888c55558 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:46:31 UTC) from {username='slave'} at 127.0.0.1:37800
I20260619 18:46:31.426402 9617 tablet_service.cc:1514] Processing DeleteTablet for tablet 37b1558706024e43a18ca0c888c55558 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:46:31 UTC) from {username='slave'} at 127.0.0.1:57270
I20260619 18:46:31.426548 9860 tablet_replica.cc:330] T 37b1558706024e43a18ca0c888c55558 P ea46bc7bc8ad489b90e8bcb3eba7b3ad: stopping tablet replica
I20260619 18:46:31.426541 9859 tablet_replica.cc:330] T 37b1558706024e43a18ca0c888c55558 P 4d2454d28b9b4af29710e399124afac7: stopping tablet replica
I20260619 18:46:31.426853 9859 raft_consensus.cc:2229] T 37b1558706024e43a18ca0c888c55558 P 4d2454d28b9b4af29710e399124afac7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:46:31.426853 9860 raft_consensus.cc:2229] T 37b1558706024e43a18ca0c888c55558 P ea46bc7bc8ad489b90e8bcb3eba7b3ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:46:31.427167 9860 raft_consensus.cc:2258] T 37b1558706024e43a18ca0c888c55558 P ea46bc7bc8ad489b90e8bcb3eba7b3ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:46:31.427176 9859 raft_consensus.cc:2258] T 37b1558706024e43a18ca0c888c55558 P 4d2454d28b9b4af29710e399124afac7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:46:31.427611 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 9424
I20260619 18:46:31.429142 9860 ts_tablet_manager.cc:1905] T 37b1558706024e43a18ca0c888c55558 P ea46bc7bc8ad489b90e8bcb3eba7b3ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:46:31.431491 9860 ts_tablet_manager.cc:1918] T 37b1558706024e43a18ca0c888c55558 P ea46bc7bc8ad489b90e8bcb3eba7b3ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:46:31.431591 9860 log.cc:1196] T 37b1558706024e43a18ca0c888c55558 P ea46bc7bc8ad489b90e8bcb3eba7b3ad: Deleting WAL directory at /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894542572984-22077-0/minicluster-data/ts-2/wal/wals/37b1558706024e43a18ca0c888c55558
I20260619 18:46:31.431953 9860 ts_tablet_manager.cc:1939] T 37b1558706024e43a18ca0c888c55558 P ea46bc7bc8ad489b90e8bcb3eba7b3ad: Deleting consensus metadata
I20260619 18:46:31.433233 9861 tablet_replica.cc:330] T 37b1558706024e43a18ca0c888c55558 P 5997d90e4857497ba50c7a29a2456a1b: stopping tablet replica
I20260619 18:46:31.433319 9348 catalog_manager.cc:4841] TS ea46bc7bc8ad489b90e8bcb3eba7b3ad (127.21.143.67:41457): tablet 37b1558706024e43a18ca0c888c55558 (table test-table [id=a48959ff1e284c2798394f7d62275698]) successfully deleted
I20260619 18:46:31.433568 9861 raft_consensus.cc:2229] T 37b1558706024e43a18ca0c888c55558 P 5997d90e4857497ba50c7a29a2456a1b [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:46:31.433971 9861 raft_consensus.cc:2258] T 37b1558706024e43a18ca0c888c55558 P 5997d90e4857497ba50c7a29a2456a1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:46:31.435103 9861 ts_tablet_manager.cc:1905] T 37b1558706024e43a18ca0c888c55558 P 5997d90e4857497ba50c7a29a2456a1b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 18:46:31.436264 9351 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.143.65:36603 (error 108)
I20260619 18:46:31.437019 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 9555
I20260619 18:46:31.437466 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:31.436936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.65:36603 (local address 127.0.0.1:47880)
0619 18:46:31.437293 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:31.437362 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.65:36603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":52,"threads_started":1}
W20260619 18:46:31.437739 9351 catalog_manager.cc:4584] TS 4d2454d28b9b4af29710e399124afac7 (127.21.143.65:36603): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 37b1558706024e43a18ca0c888c55558: Network error: Client connection negotiation failed: client connection to 127.21.143.65:36603: connect: Connection refused (error 111)
I20260619 18:46:31.442793 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 9686
I20260619 18:46:31.443657 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:31.443044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.66:35915 (local address 127.0.0.1:57298)
0619 18:46:31.443495 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:31.443557 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.66:35915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347}
I20260619 18:46:31.448283 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 9333
2026-06-19T18:46:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1510 ms)
[----------] 6 tests from EncryptionPolicyTest (10834 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:31 dist-test-slave-mld5 krb5kdc[9866](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:31 dist-test-slave-mld5 krb5kdc[9866](info): set up 2 sockets
Jun 19 18:46:31 dist-test-slave-mld5 krb5kdc[9866](info): commencing operation
krb5kdc: starting...
W20260619 18:46:32.479492 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:46:32 dist-test-slave-mld5 krb5kdc[9866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:46:32.602370 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42547
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:32.712018 9879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:32.712312 9879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:32.712389 9879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:32.716897 9879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:32.716995 9879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:32.717056 9879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:32.717094 9879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:32.717125 9879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:32.722905 9879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42547
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:32.724076 9879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:32.725127 9879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:32.733556 9884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:32.733546 9887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:32.733546 9885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:32.733758 9879 server_base.cc:1034] running on GCE node
I20260619 18:46:32.734344 9879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:32.735522 9879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:32.736709 9879 hybrid_clock.cc:648] HybridClock initialized: now 1781894792736700 us; error 31 us; skew 500 ppm
Jun 19 18:46:32 dist-test-slave-mld5 krb5kdc[9866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:32.739477 9879 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:46:32.740362 9879 webserver.cc:460] Webserver started at http://127.0.0.1:39351/ using document root <none> and password file <none>
I20260619 18:46:32.740875 9879 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:32.740934 9879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:32.741106 9879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:32.743057 9879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "6ebab0a02b4b4523995880970b7eb737"
format_stamp: "Formatted at 2026-06-19 18:46:32 on dist-test-slave-mld5"
server_key: "6396e057c86dc1f9951873bccb1a5635"
server_key_iv: "cc24679925daea29c9ccdfc8d4761163"
server_key_version: "encryptionkey@0"
I20260619 18:46:32.743654 9879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "6ebab0a02b4b4523995880970b7eb737"
format_stamp: "Formatted at 2026-06-19 18:46:32 on dist-test-slave-mld5"
server_key: "6396e057c86dc1f9951873bccb1a5635"
server_key_iv: "cc24679925daea29c9ccdfc8d4761163"
server_key_version: "encryptionkey@0"
I20260619 18:46:32.747239 9879 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 18:46:32.749632 9894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:32.750802 9879 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 18:46:32.750897 9879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "6ebab0a02b4b4523995880970b7eb737"
format_stamp: "Formatted at 2026-06-19 18:46:32 on dist-test-slave-mld5"
server_key: "6396e057c86dc1f9951873bccb1a5635"
server_key_iv: "cc24679925daea29c9ccdfc8d4761163"
server_key_version: "encryptionkey@0"
I20260619 18:46:32.750979 9879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:32.774381 9879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:32.777431 9879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:32.777586 9879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:32.784600 9879 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42547
I20260619 18:46:32.784614 9946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42547 every 8 connection(s)
I20260619 18:46:32.785558 9879 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:32.788420 9947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:32.788694 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 9879
I20260619 18:46:32.788828 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:46:32.789129 22077 external_mini_cluster.cc:1428] Setting key 49bcca7de247ebd3bf325996e1307c1f
I20260619 18:46:32.795094 9947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ebab0a02b4b4523995880970b7eb737: Bootstrap starting.
I20260619 18:46:32.797536 9947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ebab0a02b4b4523995880970b7eb737: Neither blocks nor log segments found. Creating new log.
Jun 19 18:46:32 dist-test-slave-mld5 krb5kdc[9866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 18:46:32.798456 9947 log.cc:824] T 00000000000000000000000000000000 P 6ebab0a02b4b4523995880970b7eb737: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:32.800928 9947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ebab0a02b4b4523995880970b7eb737: No bootstrap required, opened a new log
I20260619 18:46:32.804190 9950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:32.790467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38667 (local address 127.0.0.1:42547)
0619 18:46:32.790897 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:32.790911 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:32.790944 (+ 33us) server_negotiation.cc:407] Connection header received
0619 18:46:32.791723 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:32.791751 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:32.792213 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:32.792540 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:32.793330 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:32.794389 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:32.795249 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:32.795778 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:32.795949 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:32.798753 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:32.798761 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:32.798781 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:32.798817 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:32.800950 (+ 2133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:32.801468 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:32.801475 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:32.801481 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:32.801616 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:32.801925 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:32.801929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:32.801930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:32.802432 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:32.802625 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:32.802857 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 18:46:32.803115 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":140,"threads_started":1}
I20260619 18:46:32.805661 9947 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ebab0a02b4b4523995880970b7eb737 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ebab0a02b4b4523995880970b7eb737" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42547 } }
I20260619 18:46:32.805874 9947 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ebab0a02b4b4523995880970b7eb737 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:32.805936 9947 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ebab0a02b4b4523995880970b7eb737 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ebab0a02b4b4523995880970b7eb737, State: Initialized, Role: FOLLOWER
I20260619 18:46:32.806417 9947 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ebab0a02b4b4523995880970b7eb737 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ebab0a02b4b4523995880970b7eb737" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42547 } }
I20260619 18:46:32.806589 9947 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ebab0a02b4b4523995880970b7eb737 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:32.806658 9947 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ebab0a02b4b4523995880970b7eb737 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:32.806753 9947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ebab0a02b4b4523995880970b7eb737 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:32.807381 22077 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:32.807899 9947 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ebab0a02b4b4523995880970b7eb737 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ebab0a02b4b4523995880970b7eb737" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42547 } }
I20260619 18:46:32.808233 9947 leader_election.cc:302] T 00000000000000000000000000000000 P 6ebab0a02b4b4523995880970b7eb737 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ebab0a02b4b4523995880970b7eb737; no voters:
I20260619 18:46:32.808581 9947 leader_election.cc:288] T 00000000000000000000000000000000 P 6ebab0a02b4b4523995880970b7eb737 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:32.808691 9953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ebab0a02b4b4523995880970b7eb737 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:32.808921 9953 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ebab0a02b4b4523995880970b7eb737 [term 1 LEADER]: Becoming Leader. State: Replica: 6ebab0a02b4b4523995880970b7eb737, State: Running, Role: LEADER
I20260619 18:46:32.809258 9953 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ebab0a02b4b4523995880970b7eb737 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ebab0a02b4b4523995880970b7eb737" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42547 } }
I20260619 18:46:32.809875 9947 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ebab0a02b4b4523995880970b7eb737 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:32.810518 9953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ebab0a02b4b4523995880970b7eb737 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ebab0a02b4b4523995880970b7eb737. Latest consensus state: current_term: 1 leader_uuid: "6ebab0a02b4b4523995880970b7eb737" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ebab0a02b4b4523995880970b7eb737" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42547 } } }
I20260619 18:46:32.810729 9953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ebab0a02b4b4523995880970b7eb737 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:32.810940 9957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ebab0a02b4b4523995880970b7eb737 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ebab0a02b4b4523995880970b7eb737" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ebab0a02b4b4523995880970b7eb737" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42547 } } }
I20260619 18:46:32.811064 9957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ebab0a02b4b4523995880970b7eb737 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:32.811249 9966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:32.814388 9966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:32.818926 9950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:32.809503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37825 (local address 127.0.0.1:42547)
0619 18:46:32.809637 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:32.809641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:32.809895 (+ 254us) server_negotiation.cc:407] Connection header received
0619 18:46:32.810040 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:32.810045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:32.810094 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:32.810197 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:32.811058 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:32.811883 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:32.813047 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:32.813283 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:32.813392 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:32.815129 (+ 1737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:32.815132 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:32.815135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:32.815159 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:32.816974 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:32.817548 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:32.817555 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:32.817556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:32.817647 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:32.818107 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:32.818109 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:32.818110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:32.818272 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:32.818381 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:32.818667 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 18:46:32.818744 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":52}
I20260619 18:46:32.823021 9966 catalog_manager.cc:1347] Generated new cluster ID: 6deb310aed1e4be2b3e20d2d04b07c87
I20260619 18:46:32.823114 9966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:32.838102 9966 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:32.838958 9966 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:32.848810 9966 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ebab0a02b4b4523995880970b7eb737: Generated new TSK 0
I20260619 18:46:32.849471 9966 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:32.857833 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 9879
W20260619 18:46:32.862553 9864 connection.cc:535] client connection to 127.0.0.1:42547 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42547): Connection reset by peer (error 104)
2026-06-19T18:46:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:32 dist-test-slave-mld5 krb5kdc[9978](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:32 dist-test-slave-mld5 krb5kdc[9978](info): set up 2 sockets
Jun 19 18:46:32 dist-test-slave-mld5 krb5kdc[9978](info): commencing operation
krb5kdc: starting...
W20260619 18:46:33.895407 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:46:33 dist-test-slave-mld5 krb5kdc[9978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:46:34.010370 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42993
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:34.118180 9991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:34.118449 9991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:34.118489 9991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:34.122551 9991 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:34.122613 9991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:34.122632 9991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:34.122666 9991 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:34.122681 9991 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:34.128633 9991 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42993
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:34.129642 9991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:34.130609 9991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:34.138777 9996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:34.138808 9997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:34.138952 9999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:34.139016 9991 server_base.cc:1034] running on GCE node
I20260619 18:46:34.139626 9991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:34.140615 9991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:34.141873 9991 hybrid_clock.cc:648] HybridClock initialized: now 1781894794141838 us; error 46 us; skew 500 ppm
Jun 19 18:46:34 dist-test-slave-mld5 krb5kdc[9978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:34.144586 9991 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:46:34.145538 9991 webserver.cc:460] Webserver started at http://127.0.0.1:35031/ using document root <none> and password file <none>
I20260619 18:46:34.146117 9991 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:34.146204 9991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:34.146427 9991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:34.148559 9991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "cea51f7ee1f248d8a6e9cefec845741b"
format_stamp: "Formatted at 2026-06-19 18:46:34 on dist-test-slave-mld5"
server_key: "167ef68641da565ecba7ac592df4891e"
server_key_iv: "0508ea215b7d00c4e271418d444b8792"
server_key_version: "encryptionkey@0"
I20260619 18:46:34.149117 9991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "cea51f7ee1f248d8a6e9cefec845741b"
format_stamp: "Formatted at 2026-06-19 18:46:34 on dist-test-slave-mld5"
server_key: "167ef68641da565ecba7ac592df4891e"
server_key_iv: "0508ea215b7d00c4e271418d444b8792"
server_key_version: "encryptionkey@0"
I20260619 18:46:34.152729 9991 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:46:34.154976 10006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:34.156239 9991 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:46:34.156375 9991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "cea51f7ee1f248d8a6e9cefec845741b"
format_stamp: "Formatted at 2026-06-19 18:46:34 on dist-test-slave-mld5"
server_key: "167ef68641da565ecba7ac592df4891e"
server_key_iv: "0508ea215b7d00c4e271418d444b8792"
server_key_version: "encryptionkey@0"
I20260619 18:46:34.156478 9991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:34.182212 9991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:34.185392 9991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:34.185601 9991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:34.192970 9991 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42993
I20260619 18:46:34.192982 10058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42993 every 8 connection(s)
I20260619 18:46:34.193933 9991 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:34.196457 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 9991
I20260619 18:46:34.196599 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:46:34.196906 10059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:34.196944 22077 external_mini_cluster.cc:1428] Setting key 3c54dcac6bf07c74e18d867307dea334
I20260619 18:46:34.203979 10059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cea51f7ee1f248d8a6e9cefec845741b: Bootstrap starting.
I20260619 18:46:34.205986 10062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:34.198314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35365 (local address 127.0.0.1:42993)
0619 18:46:34.198755 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:34.198769 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:34.199028 (+ 259us) server_negotiation.cc:407] Connection header received
0619 18:46:34.199755 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:34.199773 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:34.200098 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:34.200364 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:34.200699 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:34.201642 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:34.202863 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:34.203355 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:34.203496 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:34.204363 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:34.204368 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:34.204379 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:34.204746 (+ 367us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:34.204796 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:34.204817 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:34.204914 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:34.205131 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 18:46:34.205278 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":175,"threads_started":1}
I20260619 18:46:34.206326 10059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cea51f7ee1f248d8a6e9cefec845741b: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:34.207031 10059 log.cc:824] T 00000000000000000000000000000000 P cea51f7ee1f248d8a6e9cefec845741b: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:34.208729 22077 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:34.209002 10059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cea51f7ee1f248d8a6e9cefec845741b: No bootstrap required, opened a new log
I20260619 18:46:34.213526 10059 raft_consensus.cc:348] T 00000000000000000000000000000000 P cea51f7ee1f248d8a6e9cefec845741b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea51f7ee1f248d8a6e9cefec845741b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42993 } }
I20260619 18:46:34.213793 10059 raft_consensus.cc:374] T 00000000000000000000000000000000 P cea51f7ee1f248d8a6e9cefec845741b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:34.213857 10059 raft_consensus.cc:729] T 00000000000000000000000000000000 P cea51f7ee1f248d8a6e9cefec845741b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea51f7ee1f248d8a6e9cefec845741b, State: Initialized, Role: FOLLOWER
I20260619 18:46:34.214315 10059 consensus_queue.cc:260] T 00000000000000000000000000000000 P cea51f7ee1f248d8a6e9cefec845741b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea51f7ee1f248d8a6e9cefec845741b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42993 } }
I20260619 18:46:34.214471 10059 raft_consensus.cc:388] T 00000000000000000000000000000000 P cea51f7ee1f248d8a6e9cefec845741b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:34.214546 10059 raft_consensus.cc:482] T 00000000000000000000000000000000 P cea51f7ee1f248d8a6e9cefec845741b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:34.214644 10059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cea51f7ee1f248d8a6e9cefec845741b [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:34.215142 10062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:34.210940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48217 (local address 127.0.0.1:42993)
0619 18:46:34.211058 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:34.211062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:34.211276 (+ 214us) server_negotiation.cc:407] Connection header received
0619 18:46:34.211405 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:34.211410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:34.211464 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:34.211563 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:34.211786 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:34.212533 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:34.213321 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:34.213648 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:34.213747 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:34.214659 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:34.214663 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:34.214667 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:34.214679 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:34.214707 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:34.214722 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:34.214834 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:34.214992 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 18:46:34.215024 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 18:46:34.215703 10059 raft_consensus.cc:504] T 00000000000000000000000000000000 P cea51f7ee1f248d8a6e9cefec845741b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea51f7ee1f248d8a6e9cefec845741b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42993 } }
I20260619 18:46:34.216032 10059 leader_election.cc:302] T 00000000000000000000000000000000 P cea51f7ee1f248d8a6e9cefec845741b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cea51f7ee1f248d8a6e9cefec845741b; no voters:
I20260619 18:46:34.216359 10059 leader_election.cc:288] T 00000000000000000000000000000000 P cea51f7ee1f248d8a6e9cefec845741b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:34.216454 10069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cea51f7ee1f248d8a6e9cefec845741b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:34.216681 10069 raft_consensus.cc:686] T 00000000000000000000000000000000 P cea51f7ee1f248d8a6e9cefec845741b [term 1 LEADER]: Becoming Leader. State: Replica: cea51f7ee1f248d8a6e9cefec845741b, State: Running, Role: LEADER
I20260619 18:46:34.216972 10069 consensus_queue.cc:237] T 00000000000000000000000000000000 P cea51f7ee1f248d8a6e9cefec845741b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea51f7ee1f248d8a6e9cefec845741b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42993 } }
I20260619 18:46:34.217460 10059 sys_catalog.cc:564] T 00000000000000000000000000000000 P cea51f7ee1f248d8a6e9cefec845741b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:34.218515 10071 sys_catalog.cc:455] T 00000000000000000000000000000000 P cea51f7ee1f248d8a6e9cefec845741b [sys.catalog]: SysCatalogTable state changed. Reason: New leader cea51f7ee1f248d8a6e9cefec845741b. Latest consensus state: current_term: 1 leader_uuid: "cea51f7ee1f248d8a6e9cefec845741b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea51f7ee1f248d8a6e9cefec845741b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42993 } } }
I20260619 18:46:34.218643 10071 sys_catalog.cc:458] T 00000000000000000000000000000000 P cea51f7ee1f248d8a6e9cefec845741b [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:34.218995 10070 sys_catalog.cc:455] T 00000000000000000000000000000000 P cea51f7ee1f248d8a6e9cefec845741b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cea51f7ee1f248d8a6e9cefec845741b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea51f7ee1f248d8a6e9cefec845741b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42993 } } }
I20260619 18:46:34.219064 10078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:34.219089 10070 sys_catalog.cc:458] T 00000000000000000000000000000000 P cea51f7ee1f248d8a6e9cefec845741b [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:34.221747 10078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:34.226431 10078 catalog_manager.cc:1347] Generated new cluster ID: 962c09901a024ec68724281dd3967209
I20260619 18:46:34.226498 10078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:34.243790 10078 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:34.244596 10078 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:34.248741 10078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cea51f7ee1f248d8a6e9cefec845741b: Generated new TSK 0
I20260619 18:46:34.249315 10078 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:34.256016 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 9991
W20260619 18:46:34.260741 9976 connection.cc:535] client connection to 127.0.0.1:42993 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42993): Connection reset by peer (error 104)
2026-06-19T18:46:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1398 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:34 dist-test-slave-mld5 krb5kdc[10090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:34 dist-test-slave-mld5 krb5kdc[10090](info): set up 2 sockets
Jun 19 18:46:34 dist-test-slave-mld5 krb5kdc[10090](info): commencing operation
krb5kdc: starting...
W20260619 18:46:35.303627 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:46:35 dist-test-slave-mld5 krb5kdc[10090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:46:35.419760 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46527
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:35.526518 10103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:35.526757 10103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:35.526796 10103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:35.530715 10103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:35.530766 10103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:35.530786 10103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:35.530802 10103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:35.530817 10103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:35.535802 10103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46527
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:35.536679 10103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:35.537568 10103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:35.545375 10109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:35.545431 10111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:35.545466 10108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:35.545491 10103 server_base.cc:1034] running on GCE node
I20260619 18:46:35.546214 10103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:35.547188 10103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:35.548384 10103 hybrid_clock.cc:648] HybridClock initialized: now 1781894795548365 us; error 28 us; skew 500 ppm
Jun 19 18:46:35 dist-test-slave-mld5 krb5kdc[10090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:35.551015 10103 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:46:35.551887 10103 webserver.cc:460] Webserver started at http://127.0.0.1:34711/ using document root <none> and password file <none>
I20260619 18:46:35.552413 10103 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:35.552501 10103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:35.552721 10103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:35.554648 10103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "bba67476f60f43b1a88faf4e908f2a73"
format_stamp: "Formatted at 2026-06-19 18:46:35 on dist-test-slave-mld5"
server_key: "2abb561f477d326df1917e677a4867d8"
server_key_iv: "d4928d214fe49877b8196684cfed04ff"
server_key_version: "encryptionkey@0"
I20260619 18:46:35.555213 10103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "bba67476f60f43b1a88faf4e908f2a73"
format_stamp: "Formatted at 2026-06-19 18:46:35 on dist-test-slave-mld5"
server_key: "2abb561f477d326df1917e677a4867d8"
server_key_iv: "d4928d214fe49877b8196684cfed04ff"
server_key_version: "encryptionkey@0"
I20260619 18:46:35.558739 10103 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:46:35.561157 10118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:35.562340 10103 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:46:35.562465 10103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "bba67476f60f43b1a88faf4e908f2a73"
format_stamp: "Formatted at 2026-06-19 18:46:35 on dist-test-slave-mld5"
server_key: "2abb561f477d326df1917e677a4867d8"
server_key_iv: "d4928d214fe49877b8196684cfed04ff"
server_key_version: "encryptionkey@0"
I20260619 18:46:35.562567 10103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:35.566958 10103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:35.570066 10103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:35.570231 10103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:35.576912 10103 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46527
I20260619 18:46:35.576963 10170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46527 every 8 connection(s)
I20260619 18:46:35.577795 10103 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:35.580438 10171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:35.584822 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 10103
I20260619 18:46:35.584985 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:46:35.585276 22077 external_mini_cluster.cc:1428] Setting key 00917c356d571847dbbb544d50624df2
I20260619 18:46:35.586068 10171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bba67476f60f43b1a88faf4e908f2a73: Bootstrap starting.
I20260619 18:46:35.588376 10171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bba67476f60f43b1a88faf4e908f2a73: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:35.589079 10171 log.cc:824] T 00000000000000000000000000000000 P bba67476f60f43b1a88faf4e908f2a73: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:35.590432 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:35.586577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50509 (local address 127.0.0.1:46527)
0619 18:46:35.587030 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:35.587044 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:35.587079 (+ 35us) server_negotiation.cc:407] Connection header received
0619 18:46:35.587735 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:35.587753 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:35.588024 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:35.588341 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:35.589317 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:35.589321 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:35.589331 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:35.589339 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:35.589386 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:35.589403 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:35.589517 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:35.589632 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 18:46:35.589787 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":123,"threads_started":1}
I20260619 18:46:35.590919 10171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bba67476f60f43b1a88faf4e908f2a73: No bootstrap required, opened a new log
I20260619 18:46:35.593212 22077 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:35.595010 10171 raft_consensus.cc:348] T 00000000000000000000000000000000 P bba67476f60f43b1a88faf4e908f2a73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba67476f60f43b1a88faf4e908f2a73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46527 } }
I20260619 18:46:35.595300 10171 raft_consensus.cc:374] T 00000000000000000000000000000000 P bba67476f60f43b1a88faf4e908f2a73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:35.595397 10171 raft_consensus.cc:729] T 00000000000000000000000000000000 P bba67476f60f43b1a88faf4e908f2a73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bba67476f60f43b1a88faf4e908f2a73, State: Initialized, Role: FOLLOWER
I20260619 18:46:35.595840 10171 consensus_queue.cc:260] T 00000000000000000000000000000000 P bba67476f60f43b1a88faf4e908f2a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba67476f60f43b1a88faf4e908f2a73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46527 } }
I20260619 18:46:35.596014 10171 raft_consensus.cc:388] T 00000000000000000000000000000000 P bba67476f60f43b1a88faf4e908f2a73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:35.596078 10171 raft_consensus.cc:482] T 00000000000000000000000000000000 P bba67476f60f43b1a88faf4e908f2a73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:35.596179 10171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bba67476f60f43b1a88faf4e908f2a73 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:35.596901 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:35.595252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42481 (local address 127.0.0.1:46527)
0619 18:46:35.595391 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:35.595396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:35.595667 (+ 271us) server_negotiation.cc:407] Connection header received
0619 18:46:35.595764 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:35.595767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:35.595799 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:35.595881 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:35.596571 (+ 690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:35.596574 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:35.596576 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:35.596587 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:35.596614 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:35.596627 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:35.596717 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:35.596765 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 18:46:35.596798 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 18:46:35.597275 10171 raft_consensus.cc:504] T 00000000000000000000000000000000 P bba67476f60f43b1a88faf4e908f2a73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba67476f60f43b1a88faf4e908f2a73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46527 } }
I20260619 18:46:35.597606 10171 leader_election.cc:302] T 00000000000000000000000000000000 P bba67476f60f43b1a88faf4e908f2a73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bba67476f60f43b1a88faf4e908f2a73; no voters:
I20260619 18:46:35.597925 10171 leader_election.cc:288] T 00000000000000000000000000000000 P bba67476f60f43b1a88faf4e908f2a73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:35.598013 10181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bba67476f60f43b1a88faf4e908f2a73 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:35.598245 10181 raft_consensus.cc:686] T 00000000000000000000000000000000 P bba67476f60f43b1a88faf4e908f2a73 [term 1 LEADER]: Becoming Leader. State: Replica: bba67476f60f43b1a88faf4e908f2a73, State: Running, Role: LEADER
I20260619 18:46:35.598569 10181 consensus_queue.cc:237] T 00000000000000000000000000000000 P bba67476f60f43b1a88faf4e908f2a73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba67476f60f43b1a88faf4e908f2a73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46527 } }
I20260619 18:46:35.599105 10171 sys_catalog.cc:564] T 00000000000000000000000000000000 P bba67476f60f43b1a88faf4e908f2a73 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:35.600273 10182 sys_catalog.cc:455] T 00000000000000000000000000000000 P bba67476f60f43b1a88faf4e908f2a73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bba67476f60f43b1a88faf4e908f2a73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba67476f60f43b1a88faf4e908f2a73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46527 } } }
I20260619 18:46:35.600401 10182 sys_catalog.cc:458] T 00000000000000000000000000000000 P bba67476f60f43b1a88faf4e908f2a73 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:35.600500 10183 sys_catalog.cc:455] T 00000000000000000000000000000000 P bba67476f60f43b1a88faf4e908f2a73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bba67476f60f43b1a88faf4e908f2a73. Latest consensus state: current_term: 1 leader_uuid: "bba67476f60f43b1a88faf4e908f2a73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba67476f60f43b1a88faf4e908f2a73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46527 } } }
I20260619 18:46:35.600561 10183 sys_catalog.cc:458] T 00000000000000000000000000000000 P bba67476f60f43b1a88faf4e908f2a73 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:35.601629 10190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:35.604369 10190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:35.609606 10190 catalog_manager.cc:1347] Generated new cluster ID: 61512a07afcd49f19e2b4cb259e30e09
I20260619 18:46:35.609664 10190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:35.648902 10190 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:35.649250 10190 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:35.662022 10190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bba67476f60f43b1a88faf4e908f2a73: Generated new TSK 0
I20260619 18:46:35.662828 10190 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:35.668942 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 10103
2026-06-19T18:46:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1413 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:35 dist-test-slave-mld5 krb5kdc[10202](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:35 dist-test-slave-mld5 krb5kdc[10202](info): set up 2 sockets
Jun 19 18:46:35 dist-test-slave-mld5 krb5kdc[10202](info): commencing operation
krb5kdc: starting...
W20260619 18:46:36.711526 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:46:36 dist-test-slave-mld5 krb5kdc[10202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:46:36.828326 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41603
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:36.935949 10215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:36.936249 10215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:36.936335 10215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:36.940768 10215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:36.940852 10215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:36.940888 10215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:36.940928 10215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:36.940966 10215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:36.946671 10215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:36.947808 10215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:36.948827 10215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:36.957053 10220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:36.957093 10221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:36.957077 10223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:36.957348 10215 server_base.cc:1034] running on GCE node
I20260619 18:46:36.957897 10215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:36.958917 10215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:36.960088 10215 hybrid_clock.cc:648] HybridClock initialized: now 1781894796960071 us; error 33 us; skew 500 ppm
Jun 19 18:46:36 dist-test-slave-mld5 krb5kdc[10202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:36.966852 10215 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:46:36.967785 10215 webserver.cc:460] Webserver started at http://127.0.0.1:34227/ using document root <none> and password file <none>
I20260619 18:46:36.968341 10215 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:36.968451 10215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:36.968683 10215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:36.970592 10215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "e9f3e415ee9f489dbe0b3d7986e8f44d"
format_stamp: "Formatted at 2026-06-19 18:46:36 on dist-test-slave-mld5"
server_key: "dbda2791513c07ef3ed9093b74f890a1"
server_key_iv: "682769d0722714e54a8f137a4de73a25"
server_key_version: "encryptionkey@0"
I20260619 18:46:36.971139 10215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "e9f3e415ee9f489dbe0b3d7986e8f44d"
format_stamp: "Formatted at 2026-06-19 18:46:36 on dist-test-slave-mld5"
server_key: "dbda2791513c07ef3ed9093b74f890a1"
server_key_iv: "682769d0722714e54a8f137a4de73a25"
server_key_version: "encryptionkey@0"
I20260619 18:46:36.974716 10215 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:46:36.977012 10230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:36.978138 10215 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 18:46:36.978314 10215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "e9f3e415ee9f489dbe0b3d7986e8f44d"
format_stamp: "Formatted at 2026-06-19 18:46:36 on dist-test-slave-mld5"
server_key: "dbda2791513c07ef3ed9093b74f890a1"
server_key_iv: "682769d0722714e54a8f137a4de73a25"
server_key_version: "encryptionkey@0"
I20260619 18:46:36.978433 10215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:36.990725 10215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:36.993721 10215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:36.993906 10215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:37.000937 10215 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41603
I20260619 18:46:37.000962 10282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41603 every 8 connection(s)
I20260619 18:46:37.001806 10215 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:37.004308 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 10215
I20260619 18:46:37.004422 10283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:37.004441 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:46:37.004705 22077 external_mini_cluster.cc:1428] Setting key f1f00dbb7b162dc514f323115ed2ba8b
I20260619 18:46:37.011415 10283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f3e415ee9f489dbe0b3d7986e8f44d: Bootstrap starting.
I20260619 18:46:37.013746 10283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9f3e415ee9f489dbe0b3d7986e8f44d: Neither blocks nor log segments found. Creating new log.
Jun 19 18:46:37 dist-test-slave-mld5 krb5kdc[10202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 18:46:37.014564 10283 log.cc:824] T 00000000000000000000000000000000 P e9f3e415ee9f489dbe0b3d7986e8f44d: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:37.016803 10283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f3e415ee9f489dbe0b3d7986e8f44d: No bootstrap required, opened a new log
I20260619 18:46:37.019338 10286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:37.006000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39025 (local address 127.0.0.1:41603)
0619 18:46:37.006414 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:37.006431 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:37.006473 (+ 42us) server_negotiation.cc:407] Connection header received
0619 18:46:37.007122 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:37.007144 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:37.007591 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:37.007960 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:37.008586 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:37.010058 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:37.010989 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:37.011685 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:37.011842 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:37.014395 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:37.014403 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:37.014422 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:37.014457 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:37.016396 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:37.016790 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:37.016795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:37.016800 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:37.016944 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:37.017194 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:37.017197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:37.017199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:37.017687 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:37.017877 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:37.018063 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 18:46:37.018361 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":114,"threads_started":1}
I20260619 18:46:37.021174 10283 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9f3e415ee9f489dbe0b3d7986e8f44d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f3e415ee9f489dbe0b3d7986e8f44d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41603 } }
I20260619 18:46:37.021406 10283 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9f3e415ee9f489dbe0b3d7986e8f44d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:37.021482 10283 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9f3e415ee9f489dbe0b3d7986e8f44d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9f3e415ee9f489dbe0b3d7986e8f44d, State: Initialized, Role: FOLLOWER
I20260619 18:46:37.022120 22077 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:37.022074 10283 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9f3e415ee9f489dbe0b3d7986e8f44d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f3e415ee9f489dbe0b3d7986e8f44d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41603 } }
I20260619 18:46:37.022284 10283 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9f3e415ee9f489dbe0b3d7986e8f44d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:37.022365 10283 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9f3e415ee9f489dbe0b3d7986e8f44d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:37.022460 10283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9f3e415ee9f489dbe0b3d7986e8f44d [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:37.023361 10283 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9f3e415ee9f489dbe0b3d7986e8f44d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f3e415ee9f489dbe0b3d7986e8f44d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41603 } }
I20260619 18:46:37.023705 10283 leader_election.cc:302] T 00000000000000000000000000000000 P e9f3e415ee9f489dbe0b3d7986e8f44d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9f3e415ee9f489dbe0b3d7986e8f44d; no voters:
I20260619 18:46:37.024030 10283 leader_election.cc:288] T 00000000000000000000000000000000 P e9f3e415ee9f489dbe0b3d7986e8f44d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:37.024140 10292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9f3e415ee9f489dbe0b3d7986e8f44d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:37.024379 10292 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9f3e415ee9f489dbe0b3d7986e8f44d [term 1 LEADER]: Becoming Leader. State: Replica: e9f3e415ee9f489dbe0b3d7986e8f44d, State: Running, Role: LEADER
I20260619 18:46:37.024694 10292 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9f3e415ee9f489dbe0b3d7986e8f44d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f3e415ee9f489dbe0b3d7986e8f44d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41603 } }
I20260619 18:46:37.025556 10283 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9f3e415ee9f489dbe0b3d7986e8f44d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:37.025740 10292 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f3e415ee9f489dbe0b3d7986e8f44d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9f3e415ee9f489dbe0b3d7986e8f44d. Latest consensus state: current_term: 1 leader_uuid: "e9f3e415ee9f489dbe0b3d7986e8f44d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f3e415ee9f489dbe0b3d7986e8f44d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41603 } } }
I20260619 18:46:37.025928 10292 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f3e415ee9f489dbe0b3d7986e8f44d [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:37.026227 10294 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f3e415ee9f489dbe0b3d7986e8f44d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9f3e415ee9f489dbe0b3d7986e8f44d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f3e415ee9f489dbe0b3d7986e8f44d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41603 } } }
I20260619 18:46:37.026320 10294 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f3e415ee9f489dbe0b3d7986e8f44d [sys.catalog]: This master's current role is: LEADER
W20260619 18:46:37.030869 10309 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e9f3e415ee9f489dbe0b3d7986e8f44d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:46:37.030959 10309 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:46:37.031065 10303 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:37.031971 10303 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:37.034186 10286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:37.024432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40281 (local address 127.0.0.1:41603)
0619 18:46:37.024529 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:37.024535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:37.024579 (+ 44us) server_negotiation.cc:407] Connection header received
0619 18:46:37.024753 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:37.024756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:37.024798 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:37.024844 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:37.025672 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:37.026979 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:37.028623 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:37.028873 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:37.028953 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:37.030367 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:37.030372 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:37.030375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:37.030398 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:37.032054 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:37.032540 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:37.032544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:37.032546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:37.032631 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:37.032969 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:37.032975 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:37.032977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:37.033183 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:37.033306 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:37.033903 (+ 597us) server_negotiation.cc:299] Negotiation successful
0619 18:46:37.034014 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 18:46:37.037274 10303 catalog_manager.cc:1347] Generated new cluster ID: 31920f3f7b404e83a6e012185928c61c
I20260619 18:46:37.037376 10303 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:37.052737 10303 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:37.053618 10303 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:37.059345 10303 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9f3e415ee9f489dbe0b3d7986e8f44d: Generated new TSK 0
I20260619 18:46:37.060011 10303 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:37.068684 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 10215
2026-06-19T18:46:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1401 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:37 dist-test-slave-mld5 krb5kdc[10314](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:37 dist-test-slave-mld5 krb5kdc[10314](info): set up 2 sockets
Jun 19 18:46:37 dist-test-slave-mld5 krb5kdc[10314](info): commencing operation
krb5kdc: starting...
W20260619 18:46:38.095542 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:46:38 dist-test-slave-mld5 krb5kdc[10314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:46:38.212107 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34257
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:38.321058 10327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:38.321354 10327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:38.321431 10327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:38.325377 10327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:38.325471 10327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:38.325510 10327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:38.325559 10327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:38.325598 10327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:38.330855 10327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:38.331912 10327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:38.332906 10327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 18:46:38.341087 10327 server_base.cc:1034] running on GCE node
W20260619 18:46:38.341025 10332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:38.341025 10333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:38.341044 10335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:38.341806 10327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:38.342839 10327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:38.343981 10327 hybrid_clock.cc:648] HybridClock initialized: now 1781894798343969 us; error 23 us; skew 500 ppm
Jun 19 18:46:38 dist-test-slave-mld5 krb5kdc[10314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:38.346777 10327 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:46:38.347741 10327 webserver.cc:460] Webserver started at http://127.0.0.1:40041/ using document root <none> and password file <none>
I20260619 18:46:38.348289 10327 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:38.348349 10327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:38.348583 10327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:38.350570 10327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "703b324ea5934ca391cea922ba55fca8"
format_stamp: "Formatted at 2026-06-19 18:46:38 on dist-test-slave-mld5"
server_key: "ba8c91b31d6c8529f2399538328981e0"
server_key_iv: "243860d750a57c915ab7e42c0aa966ca"
server_key_version: "encryptionkey@0"
I20260619 18:46:38.351121 10327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "703b324ea5934ca391cea922ba55fca8"
format_stamp: "Formatted at 2026-06-19 18:46:38 on dist-test-slave-mld5"
server_key: "ba8c91b31d6c8529f2399538328981e0"
server_key_iv: "243860d750a57c915ab7e42c0aa966ca"
server_key_version: "encryptionkey@0"
I20260619 18:46:38.354622 10327 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 18:46:38.356907 10342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:38.358059 10327 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:46:38.358191 10327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "703b324ea5934ca391cea922ba55fca8"
format_stamp: "Formatted at 2026-06-19 18:46:38 on dist-test-slave-mld5"
server_key: "ba8c91b31d6c8529f2399538328981e0"
server_key_iv: "243860d750a57c915ab7e42c0aa966ca"
server_key_version: "encryptionkey@0"
I20260619 18:46:38.358296 10327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:38.369613 10327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:38.372718 10327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:38.372920 10327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:38.379678 10327 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34257
I20260619 18:46:38.379702 10394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34257 every 8 connection(s)
I20260619 18:46:38.380594 10327 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:38.383234 10395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:38.387563 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 10327
I20260619 18:46:38.387699 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:46:38.387964 22077 external_mini_cluster.cc:1428] Setting key 90a6bb993746af03d813bf1218a3abca
I20260619 18:46:38.388764 10395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 703b324ea5934ca391cea922ba55fca8: Bootstrap starting.
I20260619 18:46:38.391367 10395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 703b324ea5934ca391cea922ba55fca8: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:38.392220 10395 log.cc:824] T 00000000000000000000000000000000 P 703b324ea5934ca391cea922ba55fca8: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:38.394172 10395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 703b324ea5934ca391cea922ba55fca8: No bootstrap required, opened a new log
I20260619 18:46:38.397703 10398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:38.389298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33727 (local address 127.0.0.1:34257)
0619 18:46:38.389693 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:38.389722 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:38.389768 (+ 46us) server_negotiation.cc:407] Connection header received
0619 18:46:38.390588 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:38.390609 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:38.391031 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:38.391409 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:38.391842 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:38.393223 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:38.394083 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:38.394758 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:38.394909 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:38.395674 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:38.395680 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:38.395696 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:38.396196 (+ 500us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:38.396261 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:38.396290 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:38.396397 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:38.396486 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 18:46:38.396687 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":255,"thread_start_us":107,"threads_started":1}
I20260619 18:46:38.398679 10395 raft_consensus.cc:348] T 00000000000000000000000000000000 P 703b324ea5934ca391cea922ba55fca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "703b324ea5934ca391cea922ba55fca8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34257 } }
I20260619 18:46:38.398941 10395 raft_consensus.cc:374] T 00000000000000000000000000000000 P 703b324ea5934ca391cea922ba55fca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:38.398998 10395 raft_consensus.cc:729] T 00000000000000000000000000000000 P 703b324ea5934ca391cea922ba55fca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 703b324ea5934ca391cea922ba55fca8, State: Initialized, Role: FOLLOWER
I20260619 18:46:38.399498 10395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 703b324ea5934ca391cea922ba55fca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "703b324ea5934ca391cea922ba55fca8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34257 } }
I20260619 18:46:38.399714 10395 raft_consensus.cc:388] T 00000000000000000000000000000000 P 703b324ea5934ca391cea922ba55fca8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:38.399783 10395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 703b324ea5934ca391cea922ba55fca8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:38.399895 10395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 703b324ea5934ca391cea922ba55fca8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:38.400532 22077 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:38.400938 10395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 703b324ea5934ca391cea922ba55fca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "703b324ea5934ca391cea922ba55fca8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34257 } }
I20260619 18:46:38.401271 10395 leader_election.cc:302] T 00000000000000000000000000000000 P 703b324ea5934ca391cea922ba55fca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 703b324ea5934ca391cea922ba55fca8; no voters:
I20260619 18:46:38.401597 10395 leader_election.cc:288] T 00000000000000000000000000000000 P 703b324ea5934ca391cea922ba55fca8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:38.401636 10400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 703b324ea5934ca391cea922ba55fca8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:38.401907 10400 raft_consensus.cc:686] T 00000000000000000000000000000000 P 703b324ea5934ca391cea922ba55fca8 [term 1 LEADER]: Becoming Leader. State: Replica: 703b324ea5934ca391cea922ba55fca8, State: Running, Role: LEADER
I20260619 18:46:38.402240 10400 consensus_queue.cc:237] T 00000000000000000000000000000000 P 703b324ea5934ca391cea922ba55fca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "703b324ea5934ca391cea922ba55fca8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34257 } }
I20260619 18:46:38.402879 10395 sys_catalog.cc:564] T 00000000000000000000000000000000 P 703b324ea5934ca391cea922ba55fca8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:38.403905 10403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 703b324ea5934ca391cea922ba55fca8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 703b324ea5934ca391cea922ba55fca8. Latest consensus state: current_term: 1 leader_uuid: "703b324ea5934ca391cea922ba55fca8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "703b324ea5934ca391cea922ba55fca8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34257 } } }
I20260619 18:46:38.404031 10403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 703b324ea5934ca391cea922ba55fca8 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:38.404376 10404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 703b324ea5934ca391cea922ba55fca8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "703b324ea5934ca391cea922ba55fca8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "703b324ea5934ca391cea922ba55fca8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34257 } } }
I20260619 18:46:38.404522 10404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 703b324ea5934ca391cea922ba55fca8 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:38.407537 10398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:38.402653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44845 (local address 127.0.0.1:34257)
0619 18:46:38.402759 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:38.402763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:38.403156 (+ 393us) server_negotiation.cc:407] Connection header received
0619 18:46:38.403280 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:38.403283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:38.403353 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:38.403428 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:38.403806 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:38.404542 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:38.405676 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:38.405990 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:38.406065 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:38.407061 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:38.407065 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:38.407067 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:38.407080 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:38.407106 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:38.407119 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:38.407161 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:38.407364 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 18:46:38.407416 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 18:46:38.408402 10421 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 703b324ea5934ca391cea922ba55fca8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:46:38.408485 10421 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:46:38.408576 10414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:38.409297 10414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:38.414777 10414 catalog_manager.cc:1347] Generated new cluster ID: b746607b1b3b4ac782569064de077e67
I20260619 18:46:38.414844 10414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:38.434091 10414 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:38.434983 10414 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:38.443212 10414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 703b324ea5934ca391cea922ba55fca8: Generated new TSK 0
I20260619 18:46:38.443892 10414 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:38.448310 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 10327
2026-06-19T18:46:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1377 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:38 dist-test-slave-mld5 krb5kdc[10426](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:38 dist-test-slave-mld5 krb5kdc[10426](info): set up 2 sockets
Jun 19 18:46:38 dist-test-slave-mld5 krb5kdc[10426](info): commencing operation
krb5kdc: starting...
W20260619 18:46:39.487510 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:46:39 dist-test-slave-mld5 krb5kdc[10426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:46:39.599987 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35345
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:39.706647 10439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:39.706909 10439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:39.706977 10439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:39.713429 10439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:39.713526 10439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:39.713563 10439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:39.713598 10439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:39.713631 10439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:39.719694 10439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:39.720728 10439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:39.722160 10439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:39.731359 10447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:39.731355 10444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:39.731545 10445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:39.731608 10439 server_base.cc:1034] running on GCE node
I20260619 18:46:39.732175 10439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:39.733122 10439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:39.734292 10439 hybrid_clock.cc:648] HybridClock initialized: now 1781894799734256 us; error 41 us; skew 500 ppm
Jun 19 18:46:39 dist-test-slave-mld5 krb5kdc[10426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894799, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:39.736843 10439 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:46:39.737715 10439 webserver.cc:460] Webserver started at http://127.0.0.1:38985/ using document root <none> and password file <none>
I20260619 18:46:39.738270 10439 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:39.738366 10439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:39.738576 10439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:39.740604 10439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "49d22a64b5564d9790ae26c7120c73f3"
format_stamp: "Formatted at 2026-06-19 18:46:39 on dist-test-slave-mld5"
server_key: "8f609700ae2f57935ca4837a23bb8f2e"
server_key_iv: "74a0dd32acc2aedc62533f1aca7d8bb2"
server_key_version: "encryptionkey@0"
I20260619 18:46:39.741166 10439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "49d22a64b5564d9790ae26c7120c73f3"
format_stamp: "Formatted at 2026-06-19 18:46:39 on dist-test-slave-mld5"
server_key: "8f609700ae2f57935ca4837a23bb8f2e"
server_key_iv: "74a0dd32acc2aedc62533f1aca7d8bb2"
server_key_version: "encryptionkey@0"
I20260619 18:46:39.744635 10439 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:46:39.746841 10454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:39.748051 10439 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:46:39.748181 10439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "49d22a64b5564d9790ae26c7120c73f3"
format_stamp: "Formatted at 2026-06-19 18:46:39 on dist-test-slave-mld5"
server_key: "8f609700ae2f57935ca4837a23bb8f2e"
server_key_iv: "74a0dd32acc2aedc62533f1aca7d8bb2"
server_key_version: "encryptionkey@0"
I20260619 18:46:39.748282 10439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:39.752535 10439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:39.755793 10439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:39.755997 10439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:39.762611 10439 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35345
I20260619 18:46:39.762637 10506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35345 every 8 connection(s)
I20260619 18:46:39.763520 10439 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:39.765285 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 10439
I20260619 18:46:39.765401 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:46:39.765689 22077 external_mini_cluster.cc:1428] Setting key a54abd2a84057db9768ea9500991a504
I20260619 18:46:39.766314 10507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:39.771011 10510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:39.767010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59725 (local address 127.0.0.1:35345)
0619 18:46:39.767427 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:39.767446 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:39.767486 (+ 40us) server_negotiation.cc:407] Connection header received
0619 18:46:39.768326 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:39.768347 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:39.768705 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:39.769053 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:39.769665 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:39.769670 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:39.769685 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:39.769708 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:39.769765 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:39.769790 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:39.769930 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:39.770004 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 18:46:39.770148 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":111,"threads_started":1}
I20260619 18:46:39.773277 10507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49d22a64b5564d9790ae26c7120c73f3: Bootstrap starting.
I20260619 18:46:39.773720 22077 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:39.775597 10507 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49d22a64b5564d9790ae26c7120c73f3: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:39.776441 10507 log.cc:824] T 00000000000000000000000000000000 P 49d22a64b5564d9790ae26c7120c73f3: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:39.777191 10510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:39.775700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58767 (local address 127.0.0.1:35345)
0619 18:46:39.775826 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:39.775833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:39.775976 (+ 143us) server_negotiation.cc:407] Connection header received
0619 18:46:39.776049 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:39.776052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:39.776082 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:39.776157 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:39.776804 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:39.776807 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:39.776812 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:39.776823 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:39.776852 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:39.776865 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:39.776950 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:39.777025 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 18:46:39.777061 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 18:46:39.778343 10507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49d22a64b5564d9790ae26c7120c73f3: No bootstrap required, opened a new log
I20260619 18:46:39.781899 10507 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49d22a64b5564d9790ae26c7120c73f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d22a64b5564d9790ae26c7120c73f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35345 } }
I20260619 18:46:39.782145 10507 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49d22a64b5564d9790ae26c7120c73f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:39.782213 10507 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49d22a64b5564d9790ae26c7120c73f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49d22a64b5564d9790ae26c7120c73f3, State: Initialized, Role: FOLLOWER
I20260619 18:46:39.782747 10507 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49d22a64b5564d9790ae26c7120c73f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d22a64b5564d9790ae26c7120c73f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35345 } }
I20260619 18:46:39.782917 10507 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49d22a64b5564d9790ae26c7120c73f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:39.782985 10507 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49d22a64b5564d9790ae26c7120c73f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:39.783082 10507 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49d22a64b5564d9790ae26c7120c73f3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:39.783972 10507 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49d22a64b5564d9790ae26c7120c73f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d22a64b5564d9790ae26c7120c73f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35345 } }
I20260619 18:46:39.784294 10507 leader_election.cc:302] T 00000000000000000000000000000000 P 49d22a64b5564d9790ae26c7120c73f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49d22a64b5564d9790ae26c7120c73f3; no voters:
I20260619 18:46:39.784641 10507 leader_election.cc:288] T 00000000000000000000000000000000 P 49d22a64b5564d9790ae26c7120c73f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:39.785032 10518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49d22a64b5564d9790ae26c7120c73f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:39.785259 10518 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49d22a64b5564d9790ae26c7120c73f3 [term 1 LEADER]: Becoming Leader. State: Replica: 49d22a64b5564d9790ae26c7120c73f3, State: Running, Role: LEADER
I20260619 18:46:39.785671 10518 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49d22a64b5564d9790ae26c7120c73f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d22a64b5564d9790ae26c7120c73f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35345 } }
I20260619 18:46:39.785971 10507 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49d22a64b5564d9790ae26c7120c73f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:39.787788 10517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49d22a64b5564d9790ae26c7120c73f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49d22a64b5564d9790ae26c7120c73f3. Latest consensus state: current_term: 1 leader_uuid: "49d22a64b5564d9790ae26c7120c73f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d22a64b5564d9790ae26c7120c73f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35345 } } }
I20260619 18:46:39.787771 10519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49d22a64b5564d9790ae26c7120c73f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49d22a64b5564d9790ae26c7120c73f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d22a64b5564d9790ae26c7120c73f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35345 } } }
I20260619 18:46:39.787918 10517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49d22a64b5564d9790ae26c7120c73f3 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:39.787918 10519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49d22a64b5564d9790ae26c7120c73f3 [sys.catalog]: This master's current role is: LEADER
W20260619 18:46:39.790510 10531 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 49d22a64b5564d9790ae26c7120c73f3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:46:39.790611 10531 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:46:39.790853 10532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:39.791564 10532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:39.797219 10532 catalog_manager.cc:1347] Generated new cluster ID: 68d0ef4720c04b20a17a0c3fa10fe445
I20260619 18:46:39.797286 10532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:39.806939 10532 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:39.807250 10532 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:39.817070 10532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49d22a64b5564d9790ae26c7120c73f3: Generated new TSK 0
I20260619 18:46:39.817690 10532 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:39.823061 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 10439
2026-06-19T18:46:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1375 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:39 dist-test-slave-mld5 krb5kdc[10538](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:39 dist-test-slave-mld5 krb5kdc[10538](info): set up 2 sockets
Jun 19 18:46:39 dist-test-slave-mld5 krb5kdc[10538](info): commencing operation
krb5kdc: starting...
W20260619 18:46:40.855486 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:46:40 dist-test-slave-mld5 krb5kdc[10538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:46:40.969352 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46343
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:41.076503 10551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:41.076759 10551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:41.076799 10551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:41.080752 10551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:41.080811 10551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:41.080832 10551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:41.080869 10551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:41.080899 10551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:41.086102 10551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:41.087157 10551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:41.088197 10551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:41.096338 10557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:41.096350 10559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:41.096369 10556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:41.096513 10551 server_base.cc:1034] running on GCE node
I20260619 18:46:41.097241 10551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:41.098328 10551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:41.099473 10551 hybrid_clock.cc:648] HybridClock initialized: now 1781894801099447 us; error 38 us; skew 500 ppm
Jun 19 18:46:41 dist-test-slave-mld5 krb5kdc[10538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:41.102002 10551 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:46:41.102844 10551 webserver.cc:460] Webserver started at http://127.0.0.1:40543/ using document root <none> and password file <none>
I20260619 18:46:41.103428 10551 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:41.103514 10551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:41.103731 10551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:41.105731 10551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "2c5683172f9b4164a37d3498e42af971"
format_stamp: "Formatted at 2026-06-19 18:46:41 on dist-test-slave-mld5"
server_key: "be7aff4667e2cbebfc425b22a21919e9"
server_key_iv: "e66370f7fd2012152db12d098846d6e8"
server_key_version: "encryptionkey@0"
I20260619 18:46:41.106276 10551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "2c5683172f9b4164a37d3498e42af971"
format_stamp: "Formatted at 2026-06-19 18:46:41 on dist-test-slave-mld5"
server_key: "be7aff4667e2cbebfc425b22a21919e9"
server_key_iv: "e66370f7fd2012152db12d098846d6e8"
server_key_version: "encryptionkey@0"
I20260619 18:46:41.109866 10551 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 18:46:41.112126 10566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:41.113266 10551 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:46:41.113394 10551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "2c5683172f9b4164a37d3498e42af971"
format_stamp: "Formatted at 2026-06-19 18:46:41 on dist-test-slave-mld5"
server_key: "be7aff4667e2cbebfc425b22a21919e9"
server_key_iv: "e66370f7fd2012152db12d098846d6e8"
server_key_version: "encryptionkey@0"
I20260619 18:46:41.113497 10551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:41.151090 10551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:41.154183 10551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:41.154387 10551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:41.161422 10551 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46343
I20260619 18:46:41.161444 10618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46343 every 8 connection(s)
I20260619 18:46:41.162395 10551 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:41.164757 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 10551
I20260619 18:46:41.164884 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:46:41.165165 22077 external_mini_cluster.cc:1428] Setting key 9450d56c4dc8e1c1d6687108883333c3
I20260619 18:46:41.165210 10619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:41.171344 10619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c5683172f9b4164a37d3498e42af971: Bootstrap starting.
I20260619 18:46:41.173816 10619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c5683172f9b4164a37d3498e42af971: Neither blocks nor log segments found. Creating new log.
Jun 19 18:46:41 dist-test-slave-mld5 krb5kdc[10538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 18:46:41.174594 10619 log.cc:824] T 00000000000000000000000000000000 P 2c5683172f9b4164a37d3498e42af971: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:41.176635 10619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c5683172f9b4164a37d3498e42af971: No bootstrap required, opened a new log
I20260619 18:46:41.179134 10622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:41.166516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.206:58879 (local address 127.0.0.1:46343)
0619 18:46:41.166940 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:41.166954 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:41.166987 (+ 33us) server_negotiation.cc:407] Connection header received
0619 18:46:41.167863 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:41.167881 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:41.168222 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:41.168539 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:41.169322 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:41.170292 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:41.171158 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:41.171728 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:41.171896 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:41.174582 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:41.174587 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:41.174599 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:41.174626 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:41.176370 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:41.176874 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:41.176880 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:41.176885 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:41.176990 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:41.177326 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:41.177329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:41.177331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:41.177719 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:41.177927 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:41.178243 (+ 316us) server_negotiation.cc:299] Negotiation successful
0619 18:46:41.178451 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":144,"threads_started":1}
I20260619 18:46:41.180351 10619 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c5683172f9b4164a37d3498e42af971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5683172f9b4164a37d3498e42af971" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46343 } }
I20260619 18:46:41.180619 10619 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c5683172f9b4164a37d3498e42af971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:41.180680 10619 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c5683172f9b4164a37d3498e42af971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c5683172f9b4164a37d3498e42af971, State: Initialized, Role: FOLLOWER
I20260619 18:46:41.181133 10619 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c5683172f9b4164a37d3498e42af971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5683172f9b4164a37d3498e42af971" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46343 } }
I20260619 18:46:41.181298 10619 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c5683172f9b4164a37d3498e42af971 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:41.181367 10619 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c5683172f9b4164a37d3498e42af971 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:41.181483 10619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c5683172f9b4164a37d3498e42af971 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:41.182022 22077 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:41.182415 10619 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c5683172f9b4164a37d3498e42af971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5683172f9b4164a37d3498e42af971" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46343 } }
I20260619 18:46:41.182739 10619 leader_election.cc:302] T 00000000000000000000000000000000 P 2c5683172f9b4164a37d3498e42af971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c5683172f9b4164a37d3498e42af971; no voters:
I20260619 18:46:41.183046 10619 leader_election.cc:288] T 00000000000000000000000000000000 P 2c5683172f9b4164a37d3498e42af971 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:41.183111 10624 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c5683172f9b4164a37d3498e42af971 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:41.183377 10624 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c5683172f9b4164a37d3498e42af971 [term 1 LEADER]: Becoming Leader. State: Replica: 2c5683172f9b4164a37d3498e42af971, State: Running, Role: LEADER
I20260619 18:46:41.183720 10624 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c5683172f9b4164a37d3498e42af971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5683172f9b4164a37d3498e42af971" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46343 } }
I20260619 18:46:41.184356 10619 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c5683172f9b4164a37d3498e42af971 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:41.185293 10629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c5683172f9b4164a37d3498e42af971 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c5683172f9b4164a37d3498e42af971" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5683172f9b4164a37d3498e42af971" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46343 } } }
I20260619 18:46:41.185565 10628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c5683172f9b4164a37d3498e42af971 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c5683172f9b4164a37d3498e42af971. Latest consensus state: current_term: 1 leader_uuid: "2c5683172f9b4164a37d3498e42af971" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5683172f9b4164a37d3498e42af971" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46343 } } }
I20260619 18:46:41.185741 10628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c5683172f9b4164a37d3498e42af971 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:41.185962 10629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c5683172f9b4164a37d3498e42af971 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:41.186626 10639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:41.189744 10639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:41.193794 10622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:41.185162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.206:43953 (local address 127.0.0.1:46343)
0619 18:46:41.185301 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:41.185308 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:41.186575 (+ 1267us) server_negotiation.cc:407] Connection header received
0619 18:46:41.186728 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:41.186733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:41.186784 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:41.186859 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:41.187681 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:41.188170 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:41.188966 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:41.189192 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:41.189292 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:41.190626 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:41.190630 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:41.190633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:41.190665 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:41.192118 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:41.192579 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:41.192583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:41.192589 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:41.192671 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:41.192984 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:41.192988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:41.192989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:41.193148 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:41.193296 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:41.193552 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 18:46:41.193653 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 18:46:41.195567 10639 catalog_manager.cc:1347] Generated new cluster ID: e2b843624d824eea840d3f3e7c5c2d1d
I20260619 18:46:41.195631 10639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:41.244333 10639 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:41.245281 10639 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:41.257131 10639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c5683172f9b4164a37d3498e42af971: Generated new TSK 0
I20260619 18:46:41.257822 10639 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:41.273742 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 10551
W20260619 18:46:41.279102 10536 connection.cc:535] client connection to 127.0.0.1:46343 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46343): Connection reset by peer (error 104)
2026-06-19T18:46:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1450 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:41 dist-test-slave-mld5 krb5kdc[10650](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:41 dist-test-slave-mld5 krb5kdc[10650](info): set up 2 sockets
Jun 19 18:46:41 dist-test-slave-mld5 krb5kdc[10650](info): commencing operation
krb5kdc: starting...
W20260619 18:46:42.304371 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:46:42 dist-test-slave-mld5 krb5kdc[10650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:46:42.418504 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39031
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:42.527418 10663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:42.527704 10663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:42.527779 10663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:42.531694 10663 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:42.531772 10663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:42.531810 10663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:42.531847 10663 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:42.531883 10663 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:42.536988 10663 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:42.538003 10663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:42.538988 10663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:42.547013 10669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:42.547075 10668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:42.547178 10663 server_base.cc:1034] running on GCE node
W20260619 18:46:42.547382 10671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:42.547940 10663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:42.548949 10663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:42.550117 10663 hybrid_clock.cc:648] HybridClock initialized: now 1781894802550096 us; error 27 us; skew 500 ppm
Jun 19 18:46:42 dist-test-slave-mld5 krb5kdc[10650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:42.557323 10663 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:46:42.558228 10663 webserver.cc:460] Webserver started at http://127.0.0.1:38201/ using document root <none> and password file <none>
I20260619 18:46:42.558800 10663 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:42.558935 10663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:42.559211 10663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:42.562160 10663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "40ed17cdc8fb4cb494fee28fca83fc8e"
format_stamp: "Formatted at 2026-06-19 18:46:42 on dist-test-slave-mld5"
server_key: "51655af4d293901e7d684adba5b22dc9"
server_key_iv: "3e9b268dde28f38d513589ddf196bca9"
server_key_version: "encryptionkey@0"
I20260619 18:46:42.562892 10663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "40ed17cdc8fb4cb494fee28fca83fc8e"
format_stamp: "Formatted at 2026-06-19 18:46:42 on dist-test-slave-mld5"
server_key: "51655af4d293901e7d684adba5b22dc9"
server_key_iv: "3e9b268dde28f38d513589ddf196bca9"
server_key_version: "encryptionkey@0"
I20260619 18:46:42.567698 10663 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 18:46:42.570358 10678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:42.571571 10663 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:46:42.571678 10663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "40ed17cdc8fb4cb494fee28fca83fc8e"
format_stamp: "Formatted at 2026-06-19 18:46:42 on dist-test-slave-mld5"
server_key: "51655af4d293901e7d684adba5b22dc9"
server_key_iv: "3e9b268dde28f38d513589ddf196bca9"
server_key_version: "encryptionkey@0"
I20260619 18:46:42.571762 10663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:42.607374 10663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:42.610728 10663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:42.610893 10663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:42.617895 10663 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39031
I20260619 18:46:42.617923 10730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39031 every 8 connection(s)
I20260619 18:46:42.618870 10663 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:42.621593 10731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:42.624899 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 10663
I20260619 18:46:42.625058 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:46:42.625332 22077 external_mini_cluster.cc:1428] Setting key 7b4f70def8b9ba34574260f18f9807e3
I20260619 18:46:42.627646 10731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40ed17cdc8fb4cb494fee28fca83fc8e: Bootstrap starting.
I20260619 18:46:42.629756 10731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40ed17cdc8fb4cb494fee28fca83fc8e: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:42.630540 10731 log.cc:824] T 00000000000000000000000000000000 P 40ed17cdc8fb4cb494fee28fca83fc8e: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:42.632318 10731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40ed17cdc8fb4cb494fee28fca83fc8e: No bootstrap required, opened a new log
I20260619 18:46:42.634392 10734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:42.626772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.206:60437 (local address 127.0.0.1:39031)
0619 18:46:42.627201 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:42.627215 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:42.627250 (+ 35us) server_negotiation.cc:407] Connection header received
0619 18:46:42.627982 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:42.628000 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:42.628340 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:42.628677 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:42.629080 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:42.630137 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:42.631054 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:42.631546 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:42.631735 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:42.632615 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:42.632620 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:42.632632 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:42.633005 (+ 373us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:42.633057 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:42.633079 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:42.633215 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:42.633364 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 18:46:42.633497 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":142,"threads_started":1}
I20260619 18:46:42.635344 10731 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40ed17cdc8fb4cb494fee28fca83fc8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ed17cdc8fb4cb494fee28fca83fc8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39031 } }
I20260619 18:46:42.635615 10731 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40ed17cdc8fb4cb494fee28fca83fc8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:42.635680 10731 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40ed17cdc8fb4cb494fee28fca83fc8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40ed17cdc8fb4cb494fee28fca83fc8e, State: Initialized, Role: FOLLOWER
I20260619 18:46:42.636108 10731 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40ed17cdc8fb4cb494fee28fca83fc8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ed17cdc8fb4cb494fee28fca83fc8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39031 } }
I20260619 18:46:42.636282 10731 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40ed17cdc8fb4cb494fee28fca83fc8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:42.636353 10731 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40ed17cdc8fb4cb494fee28fca83fc8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:42.636461 10731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40ed17cdc8fb4cb494fee28fca83fc8e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:42.637312 22077 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:42.637482 10731 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40ed17cdc8fb4cb494fee28fca83fc8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ed17cdc8fb4cb494fee28fca83fc8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39031 } }
I20260619 18:46:42.637828 10731 leader_election.cc:302] T 00000000000000000000000000000000 P 40ed17cdc8fb4cb494fee28fca83fc8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40ed17cdc8fb4cb494fee28fca83fc8e; no voters:
I20260619 18:46:42.638145 10731 leader_election.cc:288] T 00000000000000000000000000000000 P 40ed17cdc8fb4cb494fee28fca83fc8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:42.638249 10736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40ed17cdc8fb4cb494fee28fca83fc8e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:42.638456 10736 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40ed17cdc8fb4cb494fee28fca83fc8e [term 1 LEADER]: Becoming Leader. State: Replica: 40ed17cdc8fb4cb494fee28fca83fc8e, State: Running, Role: LEADER
I20260619 18:46:42.638826 10736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40ed17cdc8fb4cb494fee28fca83fc8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ed17cdc8fb4cb494fee28fca83fc8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39031 } }
I20260619 18:46:42.639465 10731 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40ed17cdc8fb4cb494fee28fca83fc8e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:42.640458 10737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40ed17cdc8fb4cb494fee28fca83fc8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40ed17cdc8fb4cb494fee28fca83fc8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ed17cdc8fb4cb494fee28fca83fc8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39031 } } }
I20260619 18:46:42.640597 10737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40ed17cdc8fb4cb494fee28fca83fc8e [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:42.640991 10738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40ed17cdc8fb4cb494fee28fca83fc8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40ed17cdc8fb4cb494fee28fca83fc8e. Latest consensus state: current_term: 1 leader_uuid: "40ed17cdc8fb4cb494fee28fca83fc8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ed17cdc8fb4cb494fee28fca83fc8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39031 } } }
I20260619 18:46:42.641147 10738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40ed17cdc8fb4cb494fee28fca83fc8e [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:42.644344 10734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:42.639523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.206:51415 (local address 127.0.0.1:39031)
0619 18:46:42.639626 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:42.639630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:42.640089 (+ 459us) server_negotiation.cc:407] Connection header received
0619 18:46:42.640284 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:42.640288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:42.640344 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:42.640401 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:42.640828 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:42.641600 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:42.642633 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:42.642855 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:42.642968 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:42.643873 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:42.643877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:42.643880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:42.643892 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:42.643918 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:42.643931 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:42.644022 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:42.644183 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 18:46:42.644214 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260619 18:46:42.644816 10757 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 40ed17cdc8fb4cb494fee28fca83fc8e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:46:42.644899 10757 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:46:42.644991 10750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:42.645720 10750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:42.651177 10750 catalog_manager.cc:1347] Generated new cluster ID: 10cfedc06dcd489eb41118850e737595
I20260619 18:46:42.651244 10750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:42.667896 10750 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:42.668814 10750 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:42.675464 10750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40ed17cdc8fb4cb494fee28fca83fc8e: Generated new TSK 0
I20260619 18:46:42.676234 10750 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:42.689893 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 10663
W20260619 18:46:42.695058 10648 connection.cc:535] client connection to 127.0.0.1:39031 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39031): Connection reset by peer (error 104)
2026-06-19T18:46:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1416 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:42 dist-test-slave-mld5 krb5kdc[10762](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:42 dist-test-slave-mld5 krb5kdc[10762](info): set up 2 sockets
Jun 19 18:46:42 dist-test-slave-mld5 krb5kdc[10762](info): commencing operation
krb5kdc: starting...
W20260619 18:46:43.731561 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:46:43 dist-test-slave-mld5 krb5kdc[10762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:46:43.847999 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33647
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:43.958163 10775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:43.958454 10775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:43.958534 10775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:43.962513 10775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:43.962602 10775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:43.962641 10775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:43.962692 10775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:43.962733 10775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:43.967947 10775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:43.968961 10775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:43.969929 10775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:43.977918 10781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:43.977918 10780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:43.977933 10783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:43.978242 10775 server_base.cc:1034] running on GCE node
I20260619 18:46:43.978869 10775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:43.979991 10775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:43.981177 10775 hybrid_clock.cc:648] HybridClock initialized: now 1781894803981143 us; error 27 us; skew 500 ppm
Jun 19 18:46:43 dist-test-slave-mld5 krb5kdc[10762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:43.983843 10775 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:46:43.984700 10775 webserver.cc:460] Webserver started at http://127.0.0.1:46321/ using document root <none> and password file <none>
I20260619 18:46:43.985188 10775 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:43.985246 10775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:43.985410 10775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:43.987279 10775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "c99676e5a63a482c8b59847e6f670b53"
format_stamp: "Formatted at 2026-06-19 18:46:43 on dist-test-slave-mld5"
server_key: "8e0b7b51fca78780c9fa0ab033433645"
server_key_iv: "4edb36281314f8b7e4ac50c5d37bc7a4"
server_key_version: "encryptionkey@0"
I20260619 18:46:43.987838 10775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "c99676e5a63a482c8b59847e6f670b53"
format_stamp: "Formatted at 2026-06-19 18:46:43 on dist-test-slave-mld5"
server_key: "8e0b7b51fca78780c9fa0ab033433645"
server_key_iv: "4edb36281314f8b7e4ac50c5d37bc7a4"
server_key_version: "encryptionkey@0"
I20260619 18:46:43.991255 10775 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260619 18:46:43.993666 10790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:43.994903 10775 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:46:43.995042 10775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "c99676e5a63a482c8b59847e6f670b53"
format_stamp: "Formatted at 2026-06-19 18:46:43 on dist-test-slave-mld5"
server_key: "8e0b7b51fca78780c9fa0ab033433645"
server_key_iv: "4edb36281314f8b7e4ac50c5d37bc7a4"
server_key_version: "encryptionkey@0"
I20260619 18:46:43.995149 10775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:43.999581 10775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:44.002233 10775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:44.002408 10775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:44.009393 10775 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33647
I20260619 18:46:44.009421 10842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33647 every 8 connection(s)
I20260619 18:46:44.010274 10775 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:44.012879 10843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:44.013808 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 10775
I20260619 18:46:44.013948 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:46:44.014237 22077 external_mini_cluster.cc:1428] Setting key a421517bd68dadaae3d0209a19691c6f
I20260619 18:46:44.019629 10846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:44.015630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.206:39165 (local address 127.0.0.1:33647)
0619 18:46:44.016035 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:44.016054 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:44.016098 (+ 44us) server_negotiation.cc:407] Connection header received
0619 18:46:44.016896 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:44.016921 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:44.017267 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:44.017634 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:44.018273 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:44.018278 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:44.018293 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:44.018317 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:44.018377 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:44.018404 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:44.018551 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:44.018620 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 18:46:44.018770 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":102,"threads_started":1}
I20260619 18:46:44.019706 10843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c99676e5a63a482c8b59847e6f670b53: Bootstrap starting.
I20260619 18:46:44.022125 10843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c99676e5a63a482c8b59847e6f670b53: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:44.022377 22077 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:44.023003 10843 log.cc:824] T 00000000000000000000000000000000 P c99676e5a63a482c8b59847e6f670b53: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:44.025179 10843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c99676e5a63a482c8b59847e6f670b53: No bootstrap required, opened a new log
I20260619 18:46:44.026221 10846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:44.024664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.206:55263 (local address 127.0.0.1:33647)
0619 18:46:44.024785 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:44.024798 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:44.024978 (+ 180us) server_negotiation.cc:407] Connection header received
0619 18:46:44.025054 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:44.025057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:44.025085 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:44.025169 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:44.025870 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:44.025874 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:44.025876 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:44.025888 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:44.025925 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:44.025937 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:44.026048 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:44.026085 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 18:46:44.026120 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 18:46:44.029018 10843 raft_consensus.cc:348] T 00000000000000000000000000000000 P c99676e5a63a482c8b59847e6f670b53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99676e5a63a482c8b59847e6f670b53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33647 } }
I20260619 18:46:44.029232 10843 raft_consensus.cc:374] T 00000000000000000000000000000000 P c99676e5a63a482c8b59847e6f670b53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:44.029273 10843 raft_consensus.cc:729] T 00000000000000000000000000000000 P c99676e5a63a482c8b59847e6f670b53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c99676e5a63a482c8b59847e6f670b53, State: Initialized, Role: FOLLOWER
I20260619 18:46:44.029613 10843 consensus_queue.cc:260] T 00000000000000000000000000000000 P c99676e5a63a482c8b59847e6f670b53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99676e5a63a482c8b59847e6f670b53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33647 } }
I20260619 18:46:44.029717 10843 raft_consensus.cc:388] T 00000000000000000000000000000000 P c99676e5a63a482c8b59847e6f670b53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:44.029760 10843 raft_consensus.cc:482] T 00000000000000000000000000000000 P c99676e5a63a482c8b59847e6f670b53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:44.029825 10843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c99676e5a63a482c8b59847e6f670b53 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:44.030582 10843 raft_consensus.cc:504] T 00000000000000000000000000000000 P c99676e5a63a482c8b59847e6f670b53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99676e5a63a482c8b59847e6f670b53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33647 } }
I20260619 18:46:44.030865 10843 leader_election.cc:302] T 00000000000000000000000000000000 P c99676e5a63a482c8b59847e6f670b53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c99676e5a63a482c8b59847e6f670b53; no voters:
I20260619 18:46:44.031091 10843 leader_election.cc:288] T 00000000000000000000000000000000 P c99676e5a63a482c8b59847e6f670b53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:44.031237 10853 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c99676e5a63a482c8b59847e6f670b53 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:44.031474 10853 raft_consensus.cc:686] T 00000000000000000000000000000000 P c99676e5a63a482c8b59847e6f670b53 [term 1 LEADER]: Becoming Leader. State: Replica: c99676e5a63a482c8b59847e6f670b53, State: Running, Role: LEADER
I20260619 18:46:44.031750 10853 consensus_queue.cc:237] T 00000000000000000000000000000000 P c99676e5a63a482c8b59847e6f670b53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99676e5a63a482c8b59847e6f670b53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33647 } }
I20260619 18:46:44.032265 10843 sys_catalog.cc:564] T 00000000000000000000000000000000 P c99676e5a63a482c8b59847e6f670b53 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:44.033252 10855 sys_catalog.cc:455] T 00000000000000000000000000000000 P c99676e5a63a482c8b59847e6f670b53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c99676e5a63a482c8b59847e6f670b53. Latest consensus state: current_term: 1 leader_uuid: "c99676e5a63a482c8b59847e6f670b53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99676e5a63a482c8b59847e6f670b53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33647 } } }
I20260619 18:46:44.033380 10855 sys_catalog.cc:458] T 00000000000000000000000000000000 P c99676e5a63a482c8b59847e6f670b53 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:44.033696 10854 sys_catalog.cc:455] T 00000000000000000000000000000000 P c99676e5a63a482c8b59847e6f670b53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c99676e5a63a482c8b59847e6f670b53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99676e5a63a482c8b59847e6f670b53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33647 } } }
I20260619 18:46:44.033763 10862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:44.033806 10854 sys_catalog.cc:458] T 00000000000000000000000000000000 P c99676e5a63a482c8b59847e6f670b53 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:44.037034 10862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:44.042181 10862 catalog_manager.cc:1347] Generated new cluster ID: c2b1fa49d8fb482fa5d66eb118151b4b
I20260619 18:46:44.042258 10862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:44.056515 10862 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:44.056841 10862 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:44.064519 10862 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c99676e5a63a482c8b59847e6f670b53: Generated new TSK 0
I20260619 18:46:44.065162 10862 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:44.073019 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 10775
2026-06-19T18:46:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1382 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:44 dist-test-slave-mld5 krb5kdc[10874](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:44 dist-test-slave-mld5 krb5kdc[10874](info): set up 2 sockets
Jun 19 18:46:44 dist-test-slave-mld5 krb5kdc[10874](info): commencing operation
krb5kdc: starting...
W20260619 18:46:45.103591 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:46:45 dist-test-slave-mld5 krb5kdc[10874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:46:45.216059 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35747
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:45.323575 10887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:45.323837 10887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:45.323915 10887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:45.327960 10887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:45.328035 10887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:45.328096 10887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:45.328142 10887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:45.328177 10887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:45.333333 10887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35747
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:45.334370 10887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:45.335460 10887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:45.343487 10893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:45.343487 10892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:45.343735 10887 server_base.cc:1034] running on GCE node
W20260619 18:46:45.343503 10895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:45.344348 10887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:45.345408 10887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:45.346562 10887 hybrid_clock.cc:648] HybridClock initialized: now 1781894805346549 us; error 25 us; skew 500 ppm
Jun 19 18:46:45 dist-test-slave-mld5 krb5kdc[10874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894805, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:45.349145 10887 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:46:45.350006 10887 webserver.cc:460] Webserver started at http://127.0.0.1:35573/ using document root <none> and password file <none>
I20260619 18:46:45.350534 10887 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:45.350642 10887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:45.350839 10887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:45.352937 10887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "4349f21d4e234614be9030d109930830"
format_stamp: "Formatted at 2026-06-19 18:46:45 on dist-test-slave-mld5"
server_key: "53d887269f70c88dbd66f274f3968888"
server_key_iv: "1473c6d05da2ed3d6b763d4dff89fcb7"
server_key_version: "encryptionkey@0"
I20260619 18:46:45.353571 10887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "4349f21d4e234614be9030d109930830"
format_stamp: "Formatted at 2026-06-19 18:46:45 on dist-test-slave-mld5"
server_key: "53d887269f70c88dbd66f274f3968888"
server_key_iv: "1473c6d05da2ed3d6b763d4dff89fcb7"
server_key_version: "encryptionkey@0"
I20260619 18:46:45.357134 10887 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260619 18:46:45.359289 10902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:45.360453 10887 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:46:45.360589 10887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "4349f21d4e234614be9030d109930830"
format_stamp: "Formatted at 2026-06-19 18:46:45 on dist-test-slave-mld5"
server_key: "53d887269f70c88dbd66f274f3968888"
server_key_iv: "1473c6d05da2ed3d6b763d4dff89fcb7"
server_key_version: "encryptionkey@0"
I20260619 18:46:45.360710 10887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:45.389505 10887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:45.392815 10887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:45.393015 10887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:45.399936 10887 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35747
I20260619 18:46:45.399963 10954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35747 every 8 connection(s)
I20260619 18:46:45.400933 10887 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:45.401527 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 10887
I20260619 18:46:45.401680 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:46:45.401978 22077 external_mini_cluster.cc:1428] Setting key 79f2ad0cb55ae2a7974cd85ed9bca2a2
I20260619 18:46:45.404075 10955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:45.409153 10955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4349f21d4e234614be9030d109930830: Bootstrap starting.
Jun 19 18:46:45 dist-test-slave-mld5 krb5kdc[10874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 18:46:45.411840 10955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4349f21d4e234614be9030d109930830: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:45.412520 10955 log.cc:824] T 00000000000000000000000000000000 P 4349f21d4e234614be9030d109930830: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:45.414566 10955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4349f21d4e234614be9030d109930830: No bootstrap required, opened a new log
I20260619 18:46:45.415961 10958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:45.403355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.206:34091 (local address 127.0.0.1:35747)
0619 18:46:45.403752 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:45.403765 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:45.403799 (+ 34us) server_negotiation.cc:407] Connection header received
0619 18:46:45.404481 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:45.404499 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:45.404822 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:45.405150 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:45.405926 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:45.406881 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:45.407766 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:45.408248 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:45.408426 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:45.411224 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:45.411229 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:45.411241 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:45.411266 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:45.413229 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:45.413770 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:45.413774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:45.413778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:45.413879 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:45.414202 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:45.414204 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:45.414205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:45.414548 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:45.414754 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:45.415027 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 18:46:45.415252 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":134,"threads_started":1}
I20260619 18:46:45.418553 10955 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4349f21d4e234614be9030d109930830 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4349f21d4e234614be9030d109930830" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35747 } }
I20260619 18:46:45.418745 22077 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:45.418780 10955 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4349f21d4e234614be9030d109930830 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:45.418833 10955 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4349f21d4e234614be9030d109930830 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4349f21d4e234614be9030d109930830, State: Initialized, Role: FOLLOWER
I20260619 18:46:45.419248 10955 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4349f21d4e234614be9030d109930830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4349f21d4e234614be9030d109930830" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35747 } }
I20260619 18:46:45.419431 10955 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4349f21d4e234614be9030d109930830 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:45.419497 10955 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4349f21d4e234614be9030d109930830 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:45.419601 10955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4349f21d4e234614be9030d109930830 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:45.420740 10955 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4349f21d4e234614be9030d109930830 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4349f21d4e234614be9030d109930830" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35747 } }
I20260619 18:46:45.421155 10955 leader_election.cc:302] T 00000000000000000000000000000000 P 4349f21d4e234614be9030d109930830 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4349f21d4e234614be9030d109930830; no voters:
I20260619 18:46:45.421540 10955 leader_election.cc:288] T 00000000000000000000000000000000 P 4349f21d4e234614be9030d109930830 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:45.421655 10965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4349f21d4e234614be9030d109930830 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:45.421867 10965 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4349f21d4e234614be9030d109930830 [term 1 LEADER]: Becoming Leader. State: Replica: 4349f21d4e234614be9030d109930830, State: Running, Role: LEADER
I20260619 18:46:45.422252 10965 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4349f21d4e234614be9030d109930830 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4349f21d4e234614be9030d109930830" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35747 } }
I20260619 18:46:45.422770 10955 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4349f21d4e234614be9030d109930830 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:45.423339 10965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4349f21d4e234614be9030d109930830 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4349f21d4e234614be9030d109930830. Latest consensus state: current_term: 1 leader_uuid: "4349f21d4e234614be9030d109930830" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4349f21d4e234614be9030d109930830" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35747 } } }
I20260619 18:46:45.423506 10965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4349f21d4e234614be9030d109930830 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:45.424064 10967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4349f21d4e234614be9030d109930830 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4349f21d4e234614be9030d109930830" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4349f21d4e234614be9030d109930830" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35747 } } }
I20260619 18:46:45.424183 10967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4349f21d4e234614be9030d109930830 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:45.425220 10973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:45.428359 10973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:45.429930 10958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:45.420945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.206:44687 (local address 127.0.0.1:35747)
0619 18:46:45.421062 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:45.421066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:45.421211 (+ 145us) server_negotiation.cc:407] Connection header received
0619 18:46:45.421300 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:45.421303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:45.421345 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:45.421427 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:45.422277 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:45.422999 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:45.424967 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:45.425291 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:45.425393 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:45.426786 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:45.426790 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:45.426792 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:45.426815 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:45.428242 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:45.428723 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:45.428726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:45.428728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:45.428805 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:45.429104 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:45.429108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:45.429109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:45.429275 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:45.429417 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:45.429685 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 18:46:45.429784 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 18:46:45.433468 10973 catalog_manager.cc:1347] Generated new cluster ID: 3ef245c29af944a3abed74b33d064301
I20260619 18:46:45.433525 10973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:45.447970 10973 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:45.449221 10973 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:45.464160 10973 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4349f21d4e234614be9030d109930830: Generated new TSK 0
I20260619 18:46:45.465131 10973 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:45.474342 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 10887
W20260619 18:46:45.479851 10872 connection.cc:535] client connection to 127.0.0.1:35747 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35747): Connection reset by peer (error 104)
2026-06-19T18:46:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1401 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:45 dist-test-slave-mld5 krb5kdc[10986](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:45 dist-test-slave-mld5 krb5kdc[10986](info): set up 2 sockets
Jun 19 18:46:45 dist-test-slave-mld5 krb5kdc[10986](info): commencing operation
krb5kdc: starting...
W20260619 18:46:46.499426 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:46:46 dist-test-slave-mld5 krb5kdc[10986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:46:46.614401 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38371
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:46.723024 10999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:46.723260 10999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:46.723392 10999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:46.727245 10999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:46.727342 10999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:46.727381 10999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:46.727398 10999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:46.727413 10999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:46.732583 10999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38371
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:46.733525 10999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:46.734521 10999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 18:46:46.742627 10999 server_base.cc:1034] running on GCE node
W20260619 18:46:46.742654 11004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:46.742654 11005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:46.742677 11007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:46.743372 10999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:46.744411 10999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:46.745594 10999 hybrid_clock.cc:648] HybridClock initialized: now 1781894806745569 us; error 44 us; skew 500 ppm
Jun 19 18:46:46 dist-test-slave-mld5 krb5kdc[10986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:46.748143 10999 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:46:46.748924 10999 webserver.cc:460] Webserver started at http://127.0.0.1:37539/ using document root <none> and password file <none>
I20260619 18:46:46.749454 10999 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:46.749536 10999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:46.749738 10999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:46.751698 10999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "e395a615f294450f8290bfc9c9113912"
format_stamp: "Formatted at 2026-06-19 18:46:46 on dist-test-slave-mld5"
server_key: "550eb6bbf3b8510aec73a4d2eae3df7a"
server_key_iv: "af24185cff000ee5fc6f90628925a1fe"
server_key_version: "encryptionkey@0"
I20260619 18:46:46.752226 10999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "e395a615f294450f8290bfc9c9113912"
format_stamp: "Formatted at 2026-06-19 18:46:46 on dist-test-slave-mld5"
server_key: "550eb6bbf3b8510aec73a4d2eae3df7a"
server_key_iv: "af24185cff000ee5fc6f90628925a1fe"
server_key_version: "encryptionkey@0"
I20260619 18:46:46.755686 10999 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:46:46.757910 11014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:46.759039 10999 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:46:46.759208 10999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "e395a615f294450f8290bfc9c9113912"
format_stamp: "Formatted at 2026-06-19 18:46:46 on dist-test-slave-mld5"
server_key: "550eb6bbf3b8510aec73a4d2eae3df7a"
server_key_iv: "af24185cff000ee5fc6f90628925a1fe"
server_key_version: "encryptionkey@0"
I20260619 18:46:46.759325 10999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:46.772090 10999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:46.774705 10999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:46.774905 10999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:46.781701 10999 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38371
I20260619 18:46:46.781723 11066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38371 every 8 connection(s)
I20260619 18:46:46.782599 10999 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:46.785259 11067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:46.789526 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 10999
I20260619 18:46:46.789664 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:46:46.789937 22077 external_mini_cluster.cc:1428] Setting key 7f249c91d9927b20c6598ef8c0c9f550
I20260619 18:46:46.791276 11067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e395a615f294450f8290bfc9c9113912: Bootstrap starting.
I20260619 18:46:46.793533 11067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e395a615f294450f8290bfc9c9113912: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:46.794284 11067 log.cc:824] T 00000000000000000000000000000000 P e395a615f294450f8290bfc9c9113912: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:46.796141 11067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e395a615f294450f8290bfc9c9113912: No bootstrap required, opened a new log
I20260619 18:46:46.799094 11070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:46.791370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.206:44015 (local address 127.0.0.1:38371)
0619 18:46:46.791803 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:46.791820 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:46.791857 (+ 37us) server_negotiation.cc:407] Connection header received
0619 18:46:46.792484 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:46.792500 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:46.792841 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:46.793223 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:46.793662 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:46.794758 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:46.795812 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:46.796335 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:46.796499 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:46.797463 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:46.797468 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:46.797481 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:46.797864 (+ 383us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:46.797917 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:46.797939 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:46.798085 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:46.798258 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 18:46:46.798394 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":134,"threads_started":1}
I20260619 18:46:46.799963 11067 raft_consensus.cc:348] T 00000000000000000000000000000000 P e395a615f294450f8290bfc9c9113912 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e395a615f294450f8290bfc9c9113912" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38371 } }
I20260619 18:46:46.800258 11067 raft_consensus.cc:374] T 00000000000000000000000000000000 P e395a615f294450f8290bfc9c9113912 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:46.800321 11067 raft_consensus.cc:729] T 00000000000000000000000000000000 P e395a615f294450f8290bfc9c9113912 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e395a615f294450f8290bfc9c9113912, State: Initialized, Role: FOLLOWER
I20260619 18:46:46.800748 11067 consensus_queue.cc:260] T 00000000000000000000000000000000 P e395a615f294450f8290bfc9c9113912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e395a615f294450f8290bfc9c9113912" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38371 } }
I20260619 18:46:46.800926 11067 raft_consensus.cc:388] T 00000000000000000000000000000000 P e395a615f294450f8290bfc9c9113912 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:46.800999 11067 raft_consensus.cc:482] T 00000000000000000000000000000000 P e395a615f294450f8290bfc9c9113912 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:46.801110 11067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e395a615f294450f8290bfc9c9113912 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:46.801965 22077 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:46.802161 11067 raft_consensus.cc:504] T 00000000000000000000000000000000 P e395a615f294450f8290bfc9c9113912 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e395a615f294450f8290bfc9c9113912" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38371 } }
I20260619 18:46:46.802510 11067 leader_election.cc:302] T 00000000000000000000000000000000 P e395a615f294450f8290bfc9c9113912 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e395a615f294450f8290bfc9c9113912; no voters:
I20260619 18:46:46.802851 11067 leader_election.cc:288] T 00000000000000000000000000000000 P e395a615f294450f8290bfc9c9113912 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:46.802938 11072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e395a615f294450f8290bfc9c9113912 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:46.803164 11072 raft_consensus.cc:686] T 00000000000000000000000000000000 P e395a615f294450f8290bfc9c9113912 [term 1 LEADER]: Becoming Leader. State: Replica: e395a615f294450f8290bfc9c9113912, State: Running, Role: LEADER
I20260619 18:46:46.803531 11072 consensus_queue.cc:237] T 00000000000000000000000000000000 P e395a615f294450f8290bfc9c9113912 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e395a615f294450f8290bfc9c9113912" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38371 } }
I20260619 18:46:46.804347 11067 sys_catalog.cc:564] T 00000000000000000000000000000000 P e395a615f294450f8290bfc9c9113912 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:46.805366 11073 sys_catalog.cc:455] T 00000000000000000000000000000000 P e395a615f294450f8290bfc9c9113912 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e395a615f294450f8290bfc9c9113912" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e395a615f294450f8290bfc9c9113912" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38371 } } }
I20260619 18:46:46.805526 11073 sys_catalog.cc:458] T 00000000000000000000000000000000 P e395a615f294450f8290bfc9c9113912 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:46.806066 11074 sys_catalog.cc:455] T 00000000000000000000000000000000 P e395a615f294450f8290bfc9c9113912 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e395a615f294450f8290bfc9c9113912. Latest consensus state: current_term: 1 leader_uuid: "e395a615f294450f8290bfc9c9113912" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e395a615f294450f8290bfc9c9113912" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38371 } } }
I20260619 18:46:46.806229 11074 sys_catalog.cc:458] T 00000000000000000000000000000000 P e395a615f294450f8290bfc9c9113912 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:46.806945 11086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:46.809201 11070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:46.804329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.206:47099 (local address 127.0.0.1:38371)
0619 18:46:46.804470 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:46.804474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:46.804808 (+ 334us) server_negotiation.cc:407] Connection header received
0619 18:46:46.804993 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:46.804996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:46.805039 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:46.805086 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:46.805510 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:46.806267 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:46.807461 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:46.807794 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:46.807908 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:46.808748 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:46.808751 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:46.808754 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:46.808767 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:46.808792 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:46.808804 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:46.808891 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:46.809050 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 18:46:46.809082 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 18:46:46.810212 11086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:46.815868 11086 catalog_manager.cc:1347] Generated new cluster ID: 1f483e4b151344d2bce469ce9dbaf811
I20260619 18:46:46.815941 11086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:46.824832 11086 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:46.825981 11086 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:46.832863 11086 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e395a615f294450f8290bfc9c9113912: Generated new TSK 0
I20260619 18:46:46.833539 11086 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:46.840013 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 10999
W20260619 18:46:46.845299 10984 connection.cc:535] client connection to 127.0.0.1:38371 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38371): Connection reset by peer (error 104)
2026-06-19T18:46:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1365 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:46 dist-test-slave-mld5 krb5kdc[11098](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:46 dist-test-slave-mld5 krb5kdc[11098](info): set up 2 sockets
Jun 19 18:46:46 dist-test-slave-mld5 krb5kdc[11098](info): commencing operation
krb5kdc: starting...
W20260619 18:46:47.871599 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:46:47 dist-test-slave-mld5 krb5kdc[11098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:46:47.984119 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34641
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:48.089897 11111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:48.090180 11111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:48.090255 11111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:48.094255 11111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:48.094341 11111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:48.094393 11111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:48.094463 11111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:48.094568 11111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:48.099718 11111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34641
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894542572984-22077-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:48.100714 11111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:48.101652 11111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:48.109588 11116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:48.109694 11111 server_base.cc:1034] running on GCE node
W20260619 18:46:48.109555 11119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:48.109562 11117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:48.110441 11111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:48.111575 11111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:48.112756 11111 hybrid_clock.cc:648] HybridClock initialized: now 1781894808112731 us; error 33 us; skew 500 ppm
Jun 19 18:46:48 dist-test-slave-mld5 krb5kdc[11098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:48.118265 11111 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:46:48.119207 11111 webserver.cc:460] Webserver started at http://127.0.0.1:39639/ using document root <none> and password file <none>
I20260619 18:46:48.119763 11111 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:48.119850 11111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:48.120067 11111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:48.121999 11111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "b832588ee3bf4865881cc9bfbfdfeac3"
format_stamp: "Formatted at 2026-06-19 18:46:48 on dist-test-slave-mld5"
server_key: "3770e2607c1a4be9c1732c0a0bd1e742"
server_key_iv: "0bd58cc76d2fa1fd135bed58c279e47d"
server_key_version: "encryptionkey@0"
I20260619 18:46:48.122557 11111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "b832588ee3bf4865881cc9bfbfdfeac3"
format_stamp: "Formatted at 2026-06-19 18:46:48 on dist-test-slave-mld5"
server_key: "3770e2607c1a4be9c1732c0a0bd1e742"
server_key_iv: "0bd58cc76d2fa1fd135bed58c279e47d"
server_key_version: "encryptionkey@0"
I20260619 18:46:48.126377 11111 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 18:46:48.128763 11126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:48.129878 11111 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:46:48.130029 11111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "b832588ee3bf4865881cc9bfbfdfeac3"
format_stamp: "Formatted at 2026-06-19 18:46:48 on dist-test-slave-mld5"
server_key: "3770e2607c1a4be9c1732c0a0bd1e742"
server_key_iv: "0bd58cc76d2fa1fd135bed58c279e47d"
server_key_version: "encryptionkey@0"
I20260619 18:46:48.130137 11111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:48.134727 11111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:48.137931 11111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:48.138124 11111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:48.145081 11111 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34641
I20260619 18:46:48.145094 11178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34641 every 8 connection(s)
I20260619 18:46:48.145974 11111 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:48.148602 11179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:48.149121 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 11111
I20260619 18:46:48.149264 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:46:48.149554 22077 external_mini_cluster.cc:1428] Setting key 1d5ac84a563061c3eb59062021fbcd68
I20260619 18:46:48.154404 11182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:48.150791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.206:34237 (local address 127.0.0.1:34641)
0619 18:46:48.151174 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:48.151196 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:48.151230 (+ 34us) server_negotiation.cc:407] Connection header received
0619 18:46:48.151957 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:48.151973 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:48.152240 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:48.152504 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:48.153257 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:48.153261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:48.153272 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:48.153282 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:48.153331 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:48.153349 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:48.153453 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:48.153568 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 18:46:48.153775 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":128,"threads_started":1}
I20260619 18:46:48.155373 11179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b832588ee3bf4865881cc9bfbfdfeac3: Bootstrap starting.
I20260619 18:46:48.157205 22077 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:48.157860 11179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b832588ee3bf4865881cc9bfbfdfeac3: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:48.158762 11179 log.cc:824] T 00000000000000000000000000000000 P b832588ee3bf4865881cc9bfbfdfeac3: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:48.161130 11179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b832588ee3bf4865881cc9bfbfdfeac3: No bootstrap required, opened a new log
I20260619 18:46:48.161154 11182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:48.159509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.206:54015 (local address 127.0.0.1:34641)
0619 18:46:48.159627 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:48.159632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:48.159801 (+ 169us) server_negotiation.cc:407] Connection header received
0619 18:46:48.159959 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:48.159961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:48.160012 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:48.160089 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:48.160757 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:48.160760 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:48.160762 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:48.160772 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:48.160797 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:48.160807 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:48.160873 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:48.160983 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 18:46:48.161011 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 18:46:48.164609 11179 raft_consensus.cc:348] T 00000000000000000000000000000000 P b832588ee3bf4865881cc9bfbfdfeac3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b832588ee3bf4865881cc9bfbfdfeac3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34641 } }
I20260619 18:46:48.164858 11179 raft_consensus.cc:374] T 00000000000000000000000000000000 P b832588ee3bf4865881cc9bfbfdfeac3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:48.164918 11179 raft_consensus.cc:729] T 00000000000000000000000000000000 P b832588ee3bf4865881cc9bfbfdfeac3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b832588ee3bf4865881cc9bfbfdfeac3, State: Initialized, Role: FOLLOWER
I20260619 18:46:48.165354 11179 consensus_queue.cc:260] T 00000000000000000000000000000000 P b832588ee3bf4865881cc9bfbfdfeac3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b832588ee3bf4865881cc9bfbfdfeac3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34641 } }
I20260619 18:46:48.165503 11179 raft_consensus.cc:388] T 00000000000000000000000000000000 P b832588ee3bf4865881cc9bfbfdfeac3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:48.165572 11179 raft_consensus.cc:482] T 00000000000000000000000000000000 P b832588ee3bf4865881cc9bfbfdfeac3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:48.165670 11179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b832588ee3bf4865881cc9bfbfdfeac3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:48.166558 11179 raft_consensus.cc:504] T 00000000000000000000000000000000 P b832588ee3bf4865881cc9bfbfdfeac3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b832588ee3bf4865881cc9bfbfdfeac3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34641 } }
I20260619 18:46:48.166899 11179 leader_election.cc:302] T 00000000000000000000000000000000 P b832588ee3bf4865881cc9bfbfdfeac3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b832588ee3bf4865881cc9bfbfdfeac3; no voters:
I20260619 18:46:48.167215 11179 leader_election.cc:288] T 00000000000000000000000000000000 P b832588ee3bf4865881cc9bfbfdfeac3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:48.167342 11189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b832588ee3bf4865881cc9bfbfdfeac3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:48.167569 11189 raft_consensus.cc:686] T 00000000000000000000000000000000 P b832588ee3bf4865881cc9bfbfdfeac3 [term 1 LEADER]: Becoming Leader. State: Replica: b832588ee3bf4865881cc9bfbfdfeac3, State: Running, Role: LEADER
I20260619 18:46:48.167917 11189 consensus_queue.cc:237] T 00000000000000000000000000000000 P b832588ee3bf4865881cc9bfbfdfeac3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b832588ee3bf4865881cc9bfbfdfeac3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34641 } }
I20260619 18:46:48.168432 11179 sys_catalog.cc:564] T 00000000000000000000000000000000 P b832588ee3bf4865881cc9bfbfdfeac3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:48.169651 11191 sys_catalog.cc:455] T 00000000000000000000000000000000 P b832588ee3bf4865881cc9bfbfdfeac3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b832588ee3bf4865881cc9bfbfdfeac3. Latest consensus state: current_term: 1 leader_uuid: "b832588ee3bf4865881cc9bfbfdfeac3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b832588ee3bf4865881cc9bfbfdfeac3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34641 } } }
I20260619 18:46:48.169679 11190 sys_catalog.cc:455] T 00000000000000000000000000000000 P b832588ee3bf4865881cc9bfbfdfeac3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b832588ee3bf4865881cc9bfbfdfeac3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b832588ee3bf4865881cc9bfbfdfeac3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34641 } } }
I20260619 18:46:48.169765 11191 sys_catalog.cc:458] T 00000000000000000000000000000000 P b832588ee3bf4865881cc9bfbfdfeac3 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:48.169770 11190 sys_catalog.cc:458] T 00000000000000000000000000000000 P b832588ee3bf4865881cc9bfbfdfeac3 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:48.170150 11198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:48.172909 11198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:48.177580 11198 catalog_manager.cc:1347] Generated new cluster ID: c60c8b04dc644667a2713667e06fbad1
I20260619 18:46:48.177639 11198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:48.198525 11198 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:48.198873 11198 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:48.214571 11198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b832588ee3bf4865881cc9bfbfdfeac3: Generated new TSK 0
I20260619 18:46:48.215334 11198 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:48.226574 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 11111
2026-06-19T18:46:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1386 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:48 dist-test-slave-mld5 krb5kdc[11210](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:48 dist-test-slave-mld5 krb5kdc[11210](info): set up 2 sockets
Jun 19 18:46:48 dist-test-slave-mld5 krb5kdc[11210](info): commencing operation
krb5kdc: starting...
W20260619 18:46:49.255540 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:46:49 dist-test-slave-mld5 krb5kdc[11210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.143.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
I20260619 18:46:49.374118 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:36803
--webserver_interface=127.21.143.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:36281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:36803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:49.481856 11223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:49.482087 11223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:49.482127 11223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:49.486028 11223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:49.486078 11223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:49.486096 11223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:49.486130 11223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:49.486145 11223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:49.491452 11223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:36281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:36803
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:36803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:49.492424 11223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:49.493381 11223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:49.501579 11231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:49.501516 11229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:49.501533 11228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:49.501930 11223 server_base.cc:1034] running on GCE node
I20260619 18:46:49.502453 11223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:49.503535 11223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:49.504689 11223 hybrid_clock.cc:648] HybridClock initialized: now 1781894809504673 us; error 32 us; skew 500 ppm
Jun 19 18:46:49 dist-test-slave-mld5 krb5kdc[11210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894809, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:49.507467 11223 init.cc:377] Logged in from keytab as kudu/127.21.143.126@KRBTEST.COM (short username kudu)
I20260619 18:46:49.508236 11223 webserver.cc:460] Webserver started at http://127.21.143.126:34317/ using document root <none> and password file <none>
I20260619 18:46:49.508725 11223 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:49.508787 11223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:49.508942 11223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:49.510875 11223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "a604d57d787341edb4a8a65412214e8b"
format_stamp: "Formatted at 2026-06-19 18:46:49 on dist-test-slave-mld5"
server_key: "5f6531f1ee904aaa52c04cb6a8b0b62e"
server_key_iv: "2f5f7c1095c95b9cd024f78b5d58cde9"
server_key_version: "encryptionkey@0"
I20260619 18:46:49.511482 11223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "a604d57d787341edb4a8a65412214e8b"
format_stamp: "Formatted at 2026-06-19 18:46:49 on dist-test-slave-mld5"
server_key: "5f6531f1ee904aaa52c04cb6a8b0b62e"
server_key_iv: "2f5f7c1095c95b9cd024f78b5d58cde9"
server_key_version: "encryptionkey@0"
I20260619 18:46:49.514925 11223 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:46:49.517226 11238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:49.518314 11223 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 18:46:49.518450 11223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "a604d57d787341edb4a8a65412214e8b"
format_stamp: "Formatted at 2026-06-19 18:46:49 on dist-test-slave-mld5"
server_key: "5f6531f1ee904aaa52c04cb6a8b0b62e"
server_key_iv: "2f5f7c1095c95b9cd024f78b5d58cde9"
server_key_version: "encryptionkey@0"
I20260619 18:46:49.518555 11223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:49.542591 11223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:49.545912 11223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:49.546121 11223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:49.553020 11223 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:36803
I20260619 18:46:49.553041 11290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:36803 every 8 connection(s)
I20260619 18:46:49.553936 11223 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:49.556514 11291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:49.559823 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 11223
I20260619 18:46:49.559949 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:46:49.560220 22077 external_mini_cluster.cc:1428] Setting key 754f1bdbc4ba608078ea669c829a9c04
I20260619 18:46:49.562289 11291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a604d57d787341edb4a8a65412214e8b: Bootstrap starting.
I20260619 18:46:49.564517 11291 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a604d57d787341edb4a8a65412214e8b: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:49.565228 11291 log.cc:824] T 00000000000000000000000000000000 P a604d57d787341edb4a8a65412214e8b: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:49.567010 11291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a604d57d787341edb4a8a65412214e8b: No bootstrap required, opened a new log
Jun 19 18:46:49 dist-test-slave-mld5 krb5kdc[11210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:46:49.570737 11291 raft_consensus.cc:348] T 00000000000000000000000000000000 P a604d57d787341edb4a8a65412214e8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a604d57d787341edb4a8a65412214e8b" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 36803 } }
I20260619 18:46:49.570964 11291 raft_consensus.cc:374] T 00000000000000000000000000000000 P a604d57d787341edb4a8a65412214e8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:49.571079 11291 raft_consensus.cc:729] T 00000000000000000000000000000000 P a604d57d787341edb4a8a65412214e8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a604d57d787341edb4a8a65412214e8b, State: Initialized, Role: FOLLOWER
I20260619 18:46:49.571584 11291 consensus_queue.cc:260] T 00000000000000000000000000000000 P a604d57d787341edb4a8a65412214e8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a604d57d787341edb4a8a65412214e8b" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 36803 } }
I20260619 18:46:49.571748 11291 raft_consensus.cc:388] T 00000000000000000000000000000000 P a604d57d787341edb4a8a65412214e8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:49.571820 11291 raft_consensus.cc:482] T 00000000000000000000000000000000 P a604d57d787341edb4a8a65412214e8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:49.571921 11291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a604d57d787341edb4a8a65412214e8b [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:49.572930 11291 raft_consensus.cc:504] T 00000000000000000000000000000000 P a604d57d787341edb4a8a65412214e8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a604d57d787341edb4a8a65412214e8b" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 36803 } }
I20260619 18:46:49.573251 11291 leader_election.cc:302] T 00000000000000000000000000000000 P a604d57d787341edb4a8a65412214e8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a604d57d787341edb4a8a65412214e8b; no voters:
I20260619 18:46:49.573556 11291 leader_election.cc:288] T 00000000000000000000000000000000 P a604d57d787341edb4a8a65412214e8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:49.573637 11296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a604d57d787341edb4a8a65412214e8b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:49.573861 11296 raft_consensus.cc:686] T 00000000000000000000000000000000 P a604d57d787341edb4a8a65412214e8b [term 1 LEADER]: Becoming Leader. State: Replica: a604d57d787341edb4a8a65412214e8b, State: Running, Role: LEADER
I20260619 18:46:49.574056 11294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:49.561611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60523 (local address 127.21.143.126:36803)
0619 18:46:49.562007 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:49.562020 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:49.562054 (+ 34us) server_negotiation.cc:407] Connection header received
0619 18:46:49.562702 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:49.562720 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:49.563038 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:49.563401 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:49.564187 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:49.565204 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:49.566099 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:49.566562 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:49.566728 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:49.569289 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:49.569294 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:49.569307 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:49.569333 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:49.570920 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:49.571513 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:49.571517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:49.571521 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:49.571619 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:49.571968 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:49.571971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:49.571972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:49.572364 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:49.572522 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:49.572848 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 18:46:49.573074 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":110,"threads_started":1}
I20260619 18:46:49.574438 11296 consensus_queue.cc:237] T 00000000000000000000000000000000 P a604d57d787341edb4a8a65412214e8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a604d57d787341edb4a8a65412214e8b" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 36803 } }
I20260619 18:46:49.574795 11291 sys_catalog.cc:564] T 00000000000000000000000000000000 P a604d57d787341edb4a8a65412214e8b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:49.576525 11297 sys_catalog.cc:455] T 00000000000000000000000000000000 P a604d57d787341edb4a8a65412214e8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a604d57d787341edb4a8a65412214e8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a604d57d787341edb4a8a65412214e8b" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 36803 } } }
I20260619 18:46:49.576653 11297 sys_catalog.cc:458] T 00000000000000000000000000000000 P a604d57d787341edb4a8a65412214e8b [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:49.577503 22077 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:49.577464 11298 sys_catalog.cc:455] T 00000000000000000000000000000000 P a604d57d787341edb4a8a65412214e8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a604d57d787341edb4a8a65412214e8b. Latest consensus state: current_term: 1 leader_uuid: "a604d57d787341edb4a8a65412214e8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a604d57d787341edb4a8a65412214e8b" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 36803 } } }
I20260619 18:46:49.577557 11298 sys_catalog.cc:458] T 00000000000000000000000000000000 P a604d57d787341edb4a8a65412214e8b [sys.catalog]: This master's current role is: LEADER
W20260619 18:46:49.580004 11311 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a604d57d787341edb4a8a65412214e8b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:46:49.580083 11311 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:46:49.580142 11310 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:49.580994 11310 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:49.587018 11310 catalog_manager.cc:1347] Generated new cluster ID: b807c1b3ac244d0b86dd6f5d806e0b89
I20260619 18:46:49.587078 11310 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:49.595919 11294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:49.579452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43283 (local address 127.21.143.126:36803)
0619 18:46:49.579560 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:49.579564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:49.583186 (+ 3622us) server_negotiation.cc:407] Connection header received
0619 18:46:49.583330 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:49.583333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:49.583381 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:49.583464 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:49.584311 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:49.585369 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:49.586187 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:49.586427 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:49.586535 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:49.592477 (+ 5942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:49.592480 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:49.592483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:49.592507 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:49.594141 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:49.594661 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:49.594664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:49.594666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:49.594743 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:49.595058 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:49.595061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:49.595062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:49.595258 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:49.595423 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:49.595675 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 18:46:49.595770 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 18:46:49.597841 11310 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:49.598664 11310 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:49.609025 11310 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a604d57d787341edb4a8a65412214e8b: Generated new TSK 0
I20260619 18:46:49.609679 11310 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:49.616858 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 11223
W20260619 18:46:49.622184 11208 connection.cc:535] client connection to 127.21.143.126:36803 recv error: Network error: failed to read from TLS socket (remote: 127.21.143.126:36803): Connection reset by peer (error 104)
2026-06-19T18:46:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1389 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:49 dist-test-slave-mld5 krb5kdc[11322](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:49 dist-test-slave-mld5 krb5kdc[11322](info): set up 2 sockets
Jun 19 18:46:49 dist-test-slave-mld5 krb5kdc[11322](info): commencing operation
krb5kdc: starting...
W20260619 18:46:50.643250 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:46:50 dist-test-slave-mld5 krb5kdc[11322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.143.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
I20260619 18:46:50.754379 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:37465
--webserver_interface=127.21.143.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:34749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:37465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:50.860491 11335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:50.860767 11335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:50.860847 11335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:50.864811 11335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:50.864890 11335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:50.864931 11335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:50.864990 11335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:50.865029 11335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:50.870149 11335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:34749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:37465
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:37465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:50.871176 11335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:50.872208 11335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:50.880818 11341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:50.880833 11340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:50.880856 11343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:50.881038 11335 server_base.cc:1034] running on GCE node
I20260619 18:46:50.881695 11335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:50.882719 11335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:50.883919 11335 hybrid_clock.cc:648] HybridClock initialized: now 1781894810883892 us; error 41 us; skew 500 ppm
Jun 19 18:46:50 dist-test-slave-mld5 krb5kdc[11322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894810, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:50.886597 11335 init.cc:377] Logged in from keytab as kudu/127.21.143.126@KRBTEST.COM (short username kudu)
I20260619 18:46:50.887449 11335 webserver.cc:460] Webserver started at http://127.21.143.126:42527/ using document root <none> and password file <none>
I20260619 18:46:50.887995 11335 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:50.888077 11335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:50.888290 11335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:50.890246 11335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "f67b89e9955b4165a9f74f6a14bf79e3"
format_stamp: "Formatted at 2026-06-19 18:46:50 on dist-test-slave-mld5"
server_key: "afbc16c60c1fcd4e191d15bb29ccc19b"
server_key_iv: "4b25e14f50955a6d4fa9f40d4a000b0c"
server_key_version: "encryptionkey@0"
I20260619 18:46:50.890774 11335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "f67b89e9955b4165a9f74f6a14bf79e3"
format_stamp: "Formatted at 2026-06-19 18:46:50 on dist-test-slave-mld5"
server_key: "afbc16c60c1fcd4e191d15bb29ccc19b"
server_key_iv: "4b25e14f50955a6d4fa9f40d4a000b0c"
server_key_version: "encryptionkey@0"
I20260619 18:46:50.894372 11335 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 18:46:50.896644 11350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:50.897744 11335 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260619 18:46:50.897867 11335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "f67b89e9955b4165a9f74f6a14bf79e3"
format_stamp: "Formatted at 2026-06-19 18:46:50 on dist-test-slave-mld5"
server_key: "afbc16c60c1fcd4e191d15bb29ccc19b"
server_key_iv: "4b25e14f50955a6d4fa9f40d4a000b0c"
server_key_version: "encryptionkey@0"
I20260619 18:46:50.897967 11335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:50.909346 11335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:50.912396 11335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:50.912581 11335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:50.918964 11335 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:37465
I20260619 18:46:50.919042 11402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:37465 every 8 connection(s)
I20260619 18:46:50.920002 11335 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:50.922574 11403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:50.928275 11403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f67b89e9955b4165a9f74f6a14bf79e3: Bootstrap starting.
I20260619 18:46:50.929630 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 11335
I20260619 18:46:50.929786 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:46:50.930051 22077 external_mini_cluster.cc:1428] Setting key 85963cec2635e76433373f9103e6ebb1
I20260619 18:46:50.930863 11403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f67b89e9955b4165a9f74f6a14bf79e3: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:50.931751 11403 log.cc:824] T 00000000000000000000000000000000 P f67b89e9955b4165a9f74f6a14bf79e3: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:50.933781 11403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f67b89e9955b4165a9f74f6a14bf79e3: No bootstrap required, opened a new log
I20260619 18:46:50.937942 11403 raft_consensus.cc:348] T 00000000000000000000000000000000 P f67b89e9955b4165a9f74f6a14bf79e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67b89e9955b4165a9f74f6a14bf79e3" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37465 } }
I20260619 18:46:50.938232 11403 raft_consensus.cc:374] T 00000000000000000000000000000000 P f67b89e9955b4165a9f74f6a14bf79e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:50.938298 11403 raft_consensus.cc:729] T 00000000000000000000000000000000 P f67b89e9955b4165a9f74f6a14bf79e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f67b89e9955b4165a9f74f6a14bf79e3, State: Initialized, Role: FOLLOWER
I20260619 18:46:50.938724 11403 consensus_queue.cc:260] T 00000000000000000000000000000000 P f67b89e9955b4165a9f74f6a14bf79e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67b89e9955b4165a9f74f6a14bf79e3" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37465 } }
I20260619 18:46:50.938874 11403 raft_consensus.cc:388] T 00000000000000000000000000000000 P f67b89e9955b4165a9f74f6a14bf79e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:50.938959 11403 raft_consensus.cc:482] T 00000000000000000000000000000000 P f67b89e9955b4165a9f74f6a14bf79e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:50.939056 11403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f67b89e9955b4165a9f74f6a14bf79e3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:50.939100 11406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:50.931165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33095 (local address 127.21.143.126:37465)
0619 18:46:50.931617 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:50.931631 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:50.931670 (+ 39us) server_negotiation.cc:407] Connection header received
0619 18:46:50.932486 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:50.932512 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:50.932832 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:50.933104 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:50.933489 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:50.934528 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:50.935775 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:50.936245 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:50.936354 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:50.937276 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:50.937281 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:50.937293 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:50.937670 (+ 377us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:50.937719 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:50.937740 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:50.937836 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:50.938046 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 18:46:50.938199 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":145,"threads_started":1}
I20260619 18:46:50.940136 11403 raft_consensus.cc:504] T 00000000000000000000000000000000 P f67b89e9955b4165a9f74f6a14bf79e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67b89e9955b4165a9f74f6a14bf79e3" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37465 } }
I20260619 18:46:50.940515 11403 leader_election.cc:302] T 00000000000000000000000000000000 P f67b89e9955b4165a9f74f6a14bf79e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f67b89e9955b4165a9f74f6a14bf79e3; no voters:
I20260619 18:46:50.940847 11403 leader_election.cc:288] T 00000000000000000000000000000000 P f67b89e9955b4165a9f74f6a14bf79e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:50.940928 11408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f67b89e9955b4165a9f74f6a14bf79e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:50.941385 11408 raft_consensus.cc:686] T 00000000000000000000000000000000 P f67b89e9955b4165a9f74f6a14bf79e3 [term 1 LEADER]: Becoming Leader. State: Replica: f67b89e9955b4165a9f74f6a14bf79e3, State: Running, Role: LEADER
I20260619 18:46:50.941735 11408 consensus_queue.cc:237] T 00000000000000000000000000000000 P f67b89e9955b4165a9f74f6a14bf79e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67b89e9955b4165a9f74f6a14bf79e3" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37465 } }
I20260619 18:46:50.942009 22077 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:50.942126 11403 sys_catalog.cc:564] T 00000000000000000000000000000000 P f67b89e9955b4165a9f74f6a14bf79e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:50.943418 11409 sys_catalog.cc:455] T 00000000000000000000000000000000 P f67b89e9955b4165a9f74f6a14bf79e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f67b89e9955b4165a9f74f6a14bf79e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67b89e9955b4165a9f74f6a14bf79e3" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37465 } } }
I20260619 18:46:50.943547 11409 sys_catalog.cc:458] T 00000000000000000000000000000000 P f67b89e9955b4165a9f74f6a14bf79e3 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:50.943910 11410 sys_catalog.cc:455] T 00000000000000000000000000000000 P f67b89e9955b4165a9f74f6a14bf79e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f67b89e9955b4165a9f74f6a14bf79e3. Latest consensus state: current_term: 1 leader_uuid: "f67b89e9955b4165a9f74f6a14bf79e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67b89e9955b4165a9f74f6a14bf79e3" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37465 } } }
I20260619 18:46:50.944052 11410 sys_catalog.cc:458] T 00000000000000000000000000000000 P f67b89e9955b4165a9f74f6a14bf79e3 [sys.catalog]: This master's current role is: LEADER
W20260619 18:46:50.947090 11427 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f67b89e9955b4165a9f74f6a14bf79e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:46:50.947175 11427 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:46:50.947284 11423 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:50.948212 11423 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:50.949393 11406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:50.944593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35157 (local address 127.21.143.126:37465)
0619 18:46:50.944708 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:50.944712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:50.945143 (+ 431us) server_negotiation.cc:407] Connection header received
0619 18:46:50.945251 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:50.945254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:50.945294 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:50.945373 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:50.945694 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:50.946416 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:50.947562 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:50.947924 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:50.948028 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:50.948890 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:50.948896 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:50.948902 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:50.948917 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:50.948943 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:50.948957 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:50.949048 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:50.949214 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 18:46:50.949248 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 18:46:50.954134 11423 catalog_manager.cc:1347] Generated new cluster ID: 9cda9e8a4abe44cbbcd489544c8a8cbe
I20260619 18:46:50.954195 11423 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:50.959991 11423 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:50.961051 11423 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:50.980501 11423 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f67b89e9955b4165a9f74f6a14bf79e3: Generated new TSK 0
I20260619 18:46:50.981230 11423 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:50.987442 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 11335
W20260619 18:46:50.991986 11320 connection.cc:535] client connection to 127.21.143.126:37465 recv error: Network error: failed to read from TLS socket (remote: 127.21.143.126:37465): Connection reset by peer (error 104)
2026-06-19T18:46:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1369 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:51 dist-test-slave-mld5 krb5kdc[11434](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:51 dist-test-slave-mld5 krb5kdc[11434](info): set up 2 sockets
Jun 19 18:46:51 dist-test-slave-mld5 krb5kdc[11434](info): commencing operation
krb5kdc: starting...
W20260619 18:46:52.015537 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:46:52 dist-test-slave-mld5 krb5kdc[11434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.143.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
I20260619 18:46:52.132360 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:40903
--webserver_interface=127.21.143.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:40483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:40903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:52.244796 11447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:52.245040 11447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:52.245080 11447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:52.249104 11447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:52.249155 11447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:52.249174 11447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:52.249228 11447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:52.249259 11447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:52.254309 11447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:40483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:40903
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:40903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:52.255322 11447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:52.256327 11447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:52.264597 11455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:52.264693 11452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:52.264688 11447 server_base.cc:1034] running on GCE node
W20260619 18:46:52.264609 11453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:52.265414 11447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:52.266466 11447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:52.267627 11447 hybrid_clock.cc:648] HybridClock initialized: now 1781894812267595 us; error 42 us; skew 500 ppm
Jun 19 18:46:52 dist-test-slave-mld5 krb5kdc[11434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894812, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:52.270226 11447 init.cc:377] Logged in from keytab as kudu/127.21.143.126@KRBTEST.COM (short username kudu)
I20260619 18:46:52.271142 11447 webserver.cc:460] Webserver started at http://127.21.143.126:35061/ using document root <none> and password file <none>
I20260619 18:46:52.271741 11447 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:52.271806 11447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:52.272014 11447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:52.274106 11447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "f3bf629bdcea4c1193d89c357720e637"
format_stamp: "Formatted at 2026-06-19 18:46:52 on dist-test-slave-mld5"
server_key: "9b546e2dc5fb1c209cadef03b6be9eb4"
server_key_iv: "f2f89ddc525f19838d472e22a7dc2b96"
server_key_version: "encryptionkey@0"
I20260619 18:46:52.274689 11447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "f3bf629bdcea4c1193d89c357720e637"
format_stamp: "Formatted at 2026-06-19 18:46:52 on dist-test-slave-mld5"
server_key: "9b546e2dc5fb1c209cadef03b6be9eb4"
server_key_iv: "f2f89ddc525f19838d472e22a7dc2b96"
server_key_version: "encryptionkey@0"
I20260619 18:46:52.278518 11447 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 18:46:52.281045 11462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:52.282269 11447 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:46:52.282408 11447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "f3bf629bdcea4c1193d89c357720e637"
format_stamp: "Formatted at 2026-06-19 18:46:52 on dist-test-slave-mld5"
server_key: "9b546e2dc5fb1c209cadef03b6be9eb4"
server_key_iv: "f2f89ddc525f19838d472e22a7dc2b96"
server_key_version: "encryptionkey@0"
I20260619 18:46:52.282522 11447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:52.287083 11447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:52.289912 11447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:52.290115 11447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:52.296998 11447 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:40903
I20260619 18:46:52.297008 11514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:40903 every 8 connection(s)
I20260619 18:46:52.297868 11447 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:52.298537 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 11447
I20260619 18:46:52.298671 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:46:52.298938 22077 external_mini_cluster.cc:1428] Setting key b17e4407efd1360ab687c5299c94b49e
I20260619 18:46:52.300726 11515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:52.303934 11518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:52.300175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50815 (local address 127.21.143.126:40903)
0619 18:46:52.300574 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:52.300594 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:52.300834 (+ 240us) server_negotiation.cc:407] Connection header received
0619 18:46:52.301528 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:52.301542 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:52.301797 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:52.302122 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:52.302803 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:52.302808 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:52.302819 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:52.302828 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:52.302874 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:52.302891 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:52.303030 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:52.303105 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 18:46:52.303247 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":144,"threads_started":1}
I20260619 18:46:52.306630 22077 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:52.306612 11515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3bf629bdcea4c1193d89c357720e637: Bootstrap starting.
I20260619 18:46:52.308924 11515 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3bf629bdcea4c1193d89c357720e637: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:52.309844 11515 log.cc:824] T 00000000000000000000000000000000 P f3bf629bdcea4c1193d89c357720e637: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:52.310289 11518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:52.308671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56017 (local address 127.21.143.126:40903)
0619 18:46:52.308777 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:52.308781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:52.308945 (+ 164us) server_negotiation.cc:407] Connection header received
0619 18:46:52.309090 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:52.309093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:52.309144 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:52.309235 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:52.309888 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:52.309892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:52.309894 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:52.309906 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:52.309935 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:52.309949 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:52.310026 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:52.310115 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 18:46:52.310156 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 18:46:52.311928 11515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3bf629bdcea4c1193d89c357720e637: No bootstrap required, opened a new log
I20260619 18:46:52.315511 11515 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3bf629bdcea4c1193d89c357720e637 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3bf629bdcea4c1193d89c357720e637" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 40903 } }
I20260619 18:46:52.315752 11515 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3bf629bdcea4c1193d89c357720e637 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:52.315831 11515 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3bf629bdcea4c1193d89c357720e637 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3bf629bdcea4c1193d89c357720e637, State: Initialized, Role: FOLLOWER
I20260619 18:46:52.316272 11515 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3bf629bdcea4c1193d89c357720e637 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3bf629bdcea4c1193d89c357720e637" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 40903 } }
I20260619 18:46:52.316413 11515 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3bf629bdcea4c1193d89c357720e637 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:52.316480 11515 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3bf629bdcea4c1193d89c357720e637 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:52.316581 11515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3bf629bdcea4c1193d89c357720e637 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:52.317444 11515 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3bf629bdcea4c1193d89c357720e637 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3bf629bdcea4c1193d89c357720e637" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 40903 } }
I20260619 18:46:52.317773 11515 leader_election.cc:302] T 00000000000000000000000000000000 P f3bf629bdcea4c1193d89c357720e637 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3bf629bdcea4c1193d89c357720e637; no voters:
I20260619 18:46:52.318073 11515 leader_election.cc:288] T 00000000000000000000000000000000 P f3bf629bdcea4c1193d89c357720e637 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:52.318181 11525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3bf629bdcea4c1193d89c357720e637 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:52.318400 11525 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3bf629bdcea4c1193d89c357720e637 [term 1 LEADER]: Becoming Leader. State: Replica: f3bf629bdcea4c1193d89c357720e637, State: Running, Role: LEADER
I20260619 18:46:52.318717 11525 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3bf629bdcea4c1193d89c357720e637 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3bf629bdcea4c1193d89c357720e637" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 40903 } }
I20260619 18:46:52.319255 11515 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3bf629bdcea4c1193d89c357720e637 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:52.320502 11527 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3bf629bdcea4c1193d89c357720e637 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3bf629bdcea4c1193d89c357720e637. Latest consensus state: current_term: 1 leader_uuid: "f3bf629bdcea4c1193d89c357720e637" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3bf629bdcea4c1193d89c357720e637" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 40903 } } }
I20260619 18:46:52.320513 11526 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3bf629bdcea4c1193d89c357720e637 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3bf629bdcea4c1193d89c357720e637" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3bf629bdcea4c1193d89c357720e637" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 40903 } } }
I20260619 18:46:52.320714 11527 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3bf629bdcea4c1193d89c357720e637 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:52.320744 11526 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3bf629bdcea4c1193d89c357720e637 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:52.321800 11534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:52.324234 11534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:52.329730 11534 catalog_manager.cc:1347] Generated new cluster ID: f743e78b74c841d6a82c39a0b3da0778
I20260619 18:46:52.329802 11534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:52.345191 11534 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:52.345492 11534 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:52.354576 11534 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3bf629bdcea4c1193d89c357720e637: Generated new TSK 0
I20260619 18:46:52.355278 11534 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:52.367576 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 11447
2026-06-19T18:46:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1380 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:52 dist-test-slave-mld5 krb5kdc[11546](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:52 dist-test-slave-mld5 krb5kdc[11546](info): set up 2 sockets
Jun 19 18:46:52 dist-test-slave-mld5 krb5kdc[11546](info): commencing operation
krb5kdc: starting...
W20260619 18:46:53.403604 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:46:53 dist-test-slave-mld5 krb5kdc[11546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.143.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
I20260619 18:46:53.529973 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:34297
--webserver_interface=127.21.143.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:41481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:34297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:53.636005 11559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:53.636252 11559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:53.636292 11559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:53.640204 11559 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:53.640252 11559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:53.640270 11559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:53.640286 11559 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:53.640300 11559 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:53.645169 11559 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:41481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:34297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:34297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:53.646126 11559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:53.647004 11559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:53.654843 11564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:53.654850 11567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:53.654857 11565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:53.655282 11559 server_base.cc:1034] running on GCE node
I20260619 18:46:53.655835 11559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:53.656816 11559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:53.657992 11559 hybrid_clock.cc:648] HybridClock initialized: now 1781894813657971 us; error 41 us; skew 500 ppm
Jun 19 18:46:53 dist-test-slave-mld5 krb5kdc[11546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894813, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:53.660583 11559 init.cc:377] Logged in from keytab as kudu/127.21.143.126@KRBTEST.COM (short username kudu)
I20260619 18:46:53.661398 11559 webserver.cc:460] Webserver started at http://127.21.143.126:36829/ using document root <none> and password file <none>
I20260619 18:46:53.661893 11559 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:53.661993 11559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:53.662204 11559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:53.664129 11559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "b34023985dbd4d60be87d65b0381f6b0"
format_stamp: "Formatted at 2026-06-19 18:46:53 on dist-test-slave-mld5"
server_key: "1391f5fb4ae6cd010931c67d5d3593e7"
server_key_iv: "b47f08f95e90d813361d67970ef2496c"
server_key_version: "encryptionkey@0"
I20260619 18:46:53.664692 11559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "b34023985dbd4d60be87d65b0381f6b0"
format_stamp: "Formatted at 2026-06-19 18:46:53 on dist-test-slave-mld5"
server_key: "1391f5fb4ae6cd010931c67d5d3593e7"
server_key_iv: "b47f08f95e90d813361d67970ef2496c"
server_key_version: "encryptionkey@0"
I20260619 18:46:53.668244 11559 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:46:53.670475 11574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:53.671725 11559 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 18:46:53.671860 11559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "b34023985dbd4d60be87d65b0381f6b0"
format_stamp: "Formatted at 2026-06-19 18:46:53 on dist-test-slave-mld5"
server_key: "1391f5fb4ae6cd010931c67d5d3593e7"
server_key_iv: "b47f08f95e90d813361d67970ef2496c"
server_key_version: "encryptionkey@0"
I20260619 18:46:53.671967 11559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:53.687247 11559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:53.690255 11559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:53.690459 11559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:53.697135 11559 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:34297
I20260619 18:46:53.697162 11626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:34297 every 8 connection(s)
I20260619 18:46:53.697999 11559 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:53.700511 11627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:53.705581 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 11559
I20260619 18:46:53.705626 11627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b34023985dbd4d60be87d65b0381f6b0: Bootstrap starting.
I20260619 18:46:53.705737 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:46:53.706015 22077 external_mini_cluster.cc:1428] Setting key 39bbdfd160cce72b231bec57771fb9cd
I20260619 18:46:53.708103 11627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b34023985dbd4d60be87d65b0381f6b0: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:53.708781 11627 log.cc:824] T 00000000000000000000000000000000 P b34023985dbd4d60be87d65b0381f6b0: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:53.710425 11627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b34023985dbd4d60be87d65b0381f6b0: No bootstrap required, opened a new log
I20260619 18:46:53.713627 11627 raft_consensus.cc:348] T 00000000000000000000000000000000 P b34023985dbd4d60be87d65b0381f6b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34023985dbd4d60be87d65b0381f6b0" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 34297 } }
I20260619 18:46:53.713893 11627 raft_consensus.cc:374] T 00000000000000000000000000000000 P b34023985dbd4d60be87d65b0381f6b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:53.713956 11627 raft_consensus.cc:729] T 00000000000000000000000000000000 P b34023985dbd4d60be87d65b0381f6b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b34023985dbd4d60be87d65b0381f6b0, State: Initialized, Role: FOLLOWER
Jun 19 18:46:53 dist-test-slave-mld5 krb5kdc[11546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:46:53.714416 11627 consensus_queue.cc:260] T 00000000000000000000000000000000 P b34023985dbd4d60be87d65b0381f6b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34023985dbd4d60be87d65b0381f6b0" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 34297 } }
I20260619 18:46:53.714587 11627 raft_consensus.cc:388] T 00000000000000000000000000000000 P b34023985dbd4d60be87d65b0381f6b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:53.714679 11627 raft_consensus.cc:482] T 00000000000000000000000000000000 P b34023985dbd4d60be87d65b0381f6b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:53.714794 11627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b34023985dbd4d60be87d65b0381f6b0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:53.715678 11627 raft_consensus.cc:504] T 00000000000000000000000000000000 P b34023985dbd4d60be87d65b0381f6b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34023985dbd4d60be87d65b0381f6b0" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 34297 } }
I20260619 18:46:53.715997 11627 leader_election.cc:302] T 00000000000000000000000000000000 P b34023985dbd4d60be87d65b0381f6b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b34023985dbd4d60be87d65b0381f6b0; no voters:
I20260619 18:46:53.716279 11627 leader_election.cc:288] T 00000000000000000000000000000000 P b34023985dbd4d60be87d65b0381f6b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:53.716408 11632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b34023985dbd4d60be87d65b0381f6b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:53.716609 11632 raft_consensus.cc:686] T 00000000000000000000000000000000 P b34023985dbd4d60be87d65b0381f6b0 [term 1 LEADER]: Becoming Leader. State: Replica: b34023985dbd4d60be87d65b0381f6b0, State: Running, Role: LEADER
I20260619 18:46:53.716980 11632 consensus_queue.cc:237] T 00000000000000000000000000000000 P b34023985dbd4d60be87d65b0381f6b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34023985dbd4d60be87d65b0381f6b0" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 34297 } }
I20260619 18:46:53.717427 11627 sys_catalog.cc:564] T 00000000000000000000000000000000 P b34023985dbd4d60be87d65b0381f6b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:53.718932 11633 sys_catalog.cc:455] T 00000000000000000000000000000000 P b34023985dbd4d60be87d65b0381f6b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b34023985dbd4d60be87d65b0381f6b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34023985dbd4d60be87d65b0381f6b0" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 34297 } } }
I20260619 18:46:53.719058 11633 sys_catalog.cc:458] T 00000000000000000000000000000000 P b34023985dbd4d60be87d65b0381f6b0 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:53.719635 11646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:53.719425 11634 sys_catalog.cc:455] T 00000000000000000000000000000000 P b34023985dbd4d60be87d65b0381f6b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b34023985dbd4d60be87d65b0381f6b0. Latest consensus state: current_term: 1 leader_uuid: "b34023985dbd4d60be87d65b0381f6b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34023985dbd4d60be87d65b0381f6b0" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 34297 } } }
I20260619 18:46:53.719790 11634 sys_catalog.cc:458] T 00000000000000000000000000000000 P b34023985dbd4d60be87d65b0381f6b0 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:53.720696 11630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:53.707292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37001 (local address 127.21.143.126:34297)
0619 18:46:53.707713 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:53.707727 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:53.707758 (+ 31us) server_negotiation.cc:407] Connection header received
0619 18:46:53.708406 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:53.708423 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:53.708734 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:53.709080 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:53.709812 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:53.710779 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:53.711724 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:53.712189 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:53.712367 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:53.714949 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:53.714954 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:53.714976 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:53.715002 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:53.717065 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:53.717511 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:53.717517 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:53.717522 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:53.717653 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:53.717943 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:53.717947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:53.717948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:53.718415 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:53.718590 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:53.719508 (+ 918us) server_negotiation.cc:299] Negotiation successful
0619 18:46:53.719803 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":118,"threads_started":1}
I20260619 18:46:53.722254 11646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:53.723601 22077 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:53.727669 11646 catalog_manager.cc:1347] Generated new cluster ID: 930d07784be34e7a80b5141225c33a67
I20260619 18:46:53.727730 11646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:53.733397 11630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:53.725707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47295 (local address 127.21.143.126:34297)
0619 18:46:53.725817 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:53.725820 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:53.726048 (+ 228us) server_negotiation.cc:407] Connection header received
0619 18:46:53.726168 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:53.726171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:53.726219 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:53.726295 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:53.727110 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:53.727864 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:53.728686 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:53.728957 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:53.729059 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:53.730231 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:53.730235 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:53.730237 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:53.730259 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:53.731780 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:53.732195 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:53.732200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:53.732201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:53.732283 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:53.732548 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:53.732553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:53.732555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:53.732730 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:53.732888 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:53.733112 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 18:46:53.733218 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:46:53.746958 11646 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:53.747987 11646 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:53.756945 11646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b34023985dbd4d60be87d65b0381f6b0: Generated new TSK 0
I20260619 18:46:53.757736 11646 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:53.766400 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 11559
2026-06-19T18:46:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1398 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:53 dist-test-slave-mld5 krb5kdc[11658](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:53 dist-test-slave-mld5 krb5kdc[11658](info): set up 2 sockets
Jun 19 18:46:53 dist-test-slave-mld5 krb5kdc[11658](info): commencing operation
krb5kdc: starting...
W20260619 18:46:54.791616 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:46:54 dist-test-slave-mld5 krb5kdc[11658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.143.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
I20260619 18:46:54.908488 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:37793
--webserver_interface=127.21.143.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:36257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:37793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:55.017401 11671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:55.017664 11671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:55.017705 11671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:55.021602 11671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:55.021662 11671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:55.021682 11671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:55.021699 11671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:55.021714 11671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:55.026944 11671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:36257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:37793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:37793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:55.027928 11671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:55.028880 11671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:55.037386 11679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:55.037390 11676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:55.037381 11677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:55.037771 11671 server_base.cc:1034] running on GCE node
I20260619 18:46:55.038352 11671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:55.039485 11671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:55.040668 11671 hybrid_clock.cc:648] HybridClock initialized: now 1781894815040641 us; error 43 us; skew 500 ppm
Jun 19 18:46:55 dist-test-slave-mld5 krb5kdc[11658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894815, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:55.043609 11671 init.cc:377] Logged in from keytab as kudu/127.21.143.126@KRBTEST.COM (short username kudu)
I20260619 18:46:55.044427 11671 webserver.cc:460] Webserver started at http://127.21.143.126:39271/ using document root <none> and password file <none>
I20260619 18:46:55.044960 11671 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:55.045017 11671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:55.045176 11671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:55.047094 11671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "dd2c27574bd64ac3bf2004ae8be9862a"
format_stamp: "Formatted at 2026-06-19 18:46:55 on dist-test-slave-mld5"
server_key: "eab9dcaf111b762630612806caec8181"
server_key_iv: "8344cd2974ba4b2fbfd1ffd82caff684"
server_key_version: "encryptionkey@0"
I20260619 18:46:55.047664 11671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "dd2c27574bd64ac3bf2004ae8be9862a"
format_stamp: "Formatted at 2026-06-19 18:46:55 on dist-test-slave-mld5"
server_key: "eab9dcaf111b762630612806caec8181"
server_key_iv: "8344cd2974ba4b2fbfd1ffd82caff684"
server_key_version: "encryptionkey@0"
I20260619 18:46:55.051196 11671 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:46:55.053628 11686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:55.054826 11671 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:46:55.054960 11671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "dd2c27574bd64ac3bf2004ae8be9862a"
format_stamp: "Formatted at 2026-06-19 18:46:55 on dist-test-slave-mld5"
server_key: "eab9dcaf111b762630612806caec8181"
server_key_iv: "8344cd2974ba4b2fbfd1ffd82caff684"
server_key_version: "encryptionkey@0"
I20260619 18:46:55.055066 11671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:55.076699 11671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:55.079972 11671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:55.080209 11671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:55.086992 11671 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:37793
I20260619 18:46:55.087069 11738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:37793 every 8 connection(s)
I20260619 18:46:55.087984 11671 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:55.090720 11739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:55.094640 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 11671
I20260619 18:46:55.094800 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:46:55.095099 22077 external_mini_cluster.cc:1428] Setting key c093f6853b315c0c1a4b022ce0c6abab
I20260619 18:46:55.096822 11739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd2c27574bd64ac3bf2004ae8be9862a: Bootstrap starting.
I20260619 18:46:55.099483 11739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd2c27574bd64ac3bf2004ae8be9862a: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:55.100440 11739 log.cc:824] T 00000000000000000000000000000000 P dd2c27574bd64ac3bf2004ae8be9862a: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:55.102712 11739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd2c27574bd64ac3bf2004ae8be9862a: No bootstrap required, opened a new log
I20260619 18:46:55.104249 11742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:55.096513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43473 (local address 127.21.143.126:37793)
0619 18:46:55.096950 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:55.096974 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:55.097019 (+ 45us) server_negotiation.cc:407] Connection header received
0619 18:46:55.097659 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:55.097676 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:55.098013 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:55.098295 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:55.098611 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:55.099689 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:55.100896 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:55.101492 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:55.101608 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:55.102483 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:55.102487 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:55.102500 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:55.102948 (+ 448us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:55.103002 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:55.103024 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:55.103098 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:55.103255 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 18:46:55.103535 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":137,"threads_started":1}
I20260619 18:46:55.106783 11739 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd2c27574bd64ac3bf2004ae8be9862a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2c27574bd64ac3bf2004ae8be9862a" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37793 } }
I20260619 18:46:55.107061 11739 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd2c27574bd64ac3bf2004ae8be9862a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:55.107153 11739 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd2c27574bd64ac3bf2004ae8be9862a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd2c27574bd64ac3bf2004ae8be9862a, State: Initialized, Role: FOLLOWER
I20260619 18:46:55.107174 22077 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:55.107678 11739 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd2c27574bd64ac3bf2004ae8be9862a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2c27574bd64ac3bf2004ae8be9862a" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37793 } }
I20260619 18:46:55.107834 11739 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd2c27574bd64ac3bf2004ae8be9862a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:55.107919 11739 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd2c27574bd64ac3bf2004ae8be9862a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:55.108019 11739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd2c27574bd64ac3bf2004ae8be9862a [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:55.109148 11739 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd2c27574bd64ac3bf2004ae8be9862a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2c27574bd64ac3bf2004ae8be9862a" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37793 } }
I20260619 18:46:55.109567 11739 leader_election.cc:302] T 00000000000000000000000000000000 P dd2c27574bd64ac3bf2004ae8be9862a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd2c27574bd64ac3bf2004ae8be9862a; no voters:
I20260619 18:46:55.109918 11739 leader_election.cc:288] T 00000000000000000000000000000000 P dd2c27574bd64ac3bf2004ae8be9862a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:55.109985 11749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd2c27574bd64ac3bf2004ae8be9862a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:55.110213 11749 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd2c27574bd64ac3bf2004ae8be9862a [term 1 LEADER]: Becoming Leader. State: Replica: dd2c27574bd64ac3bf2004ae8be9862a, State: Running, Role: LEADER
I20260619 18:46:55.110569 11749 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd2c27574bd64ac3bf2004ae8be9862a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2c27574bd64ac3bf2004ae8be9862a" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37793 } }
I20260619 18:46:55.112424 11739 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd2c27574bd64ac3bf2004ae8be9862a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:55.113060 11750 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd2c27574bd64ac3bf2004ae8be9862a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd2c27574bd64ac3bf2004ae8be9862a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2c27574bd64ac3bf2004ae8be9862a" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37793 } } }
I20260619 18:46:55.113178 11751 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd2c27574bd64ac3bf2004ae8be9862a [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd2c27574bd64ac3bf2004ae8be9862a. Latest consensus state: current_term: 1 leader_uuid: "dd2c27574bd64ac3bf2004ae8be9862a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2c27574bd64ac3bf2004ae8be9862a" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37793 } } }
I20260619 18:46:55.113231 11750 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd2c27574bd64ac3bf2004ae8be9862a [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:55.113245 11751 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd2c27574bd64ac3bf2004ae8be9862a [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:55.113601 11755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:55.114238 11742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:55.109292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33337 (local address 127.21.143.126:37793)
0619 18:46:55.109401 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:55.109405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:55.109581 (+ 176us) server_negotiation.cc:407] Connection header received
0619 18:46:55.109783 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:55.109786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:55.109824 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:55.109908 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:55.110158 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:55.110830 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:55.111961 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:55.112296 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:55.112402 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:55.113375 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:55.113380 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:55.113385 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:55.113403 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:55.113439 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:55.113460 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:55.113958 (+ 498us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:55.113992 (+ 34us) server_negotiation.cc:299] Negotiation successful
0619 18:46:55.114057 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 18:46:55.117241 11755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:55.123446 11755 catalog_manager.cc:1347] Generated new cluster ID: 0750833edec4475d83d803a3a10f2756
I20260619 18:46:55.123528 11755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:55.130795 11755 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:55.131959 11755 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:55.140463 11755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd2c27574bd64ac3bf2004ae8be9862a: Generated new TSK 0
I20260619 18:46:55.141129 11755 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:55.149474 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 11671
2026-06-19T18:46:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1383 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:55 dist-test-slave-mld5 krb5kdc[11770](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:55 dist-test-slave-mld5 krb5kdc[11770](info): set up 2 sockets
Jun 19 18:46:55 dist-test-slave-mld5 krb5kdc[11770](info): commencing operation
krb5kdc: starting...
W20260619 18:46:56.183792 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:46:56 dist-test-slave-mld5 krb5kdc[11770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.143.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
I20260619 18:46:56.300343 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:39063
--webserver_interface=127.21.143.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:43189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:39063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:56.409036 11783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:56.409380 11783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:56.409472 11783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:56.413611 11783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:56.413709 11783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:56.413748 11783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:56.413800 11783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:56.413836 11783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:56.419277 11783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:43189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:39063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:39063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:56.420357 11783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:56.421367 11783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:56.429632 11789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:56.429611 11791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:56.429606 11788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:56.429867 11783 server_base.cc:1034] running on GCE node
I20260619 18:46:56.430583 11783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:56.431736 11783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:56.432927 11783 hybrid_clock.cc:648] HybridClock initialized: now 1781894816432915 us; error 26 us; skew 500 ppm
Jun 19 18:46:56 dist-test-slave-mld5 krb5kdc[11770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894816, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:56.435748 11783 init.cc:377] Logged in from keytab as kudu/127.21.143.126@KRBTEST.COM (short username kudu)
I20260619 18:46:56.436599 11783 webserver.cc:460] Webserver started at http://127.21.143.126:37095/ using document root <none> and password file <none>
I20260619 18:46:56.437184 11783 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:56.437278 11783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:56.437510 11783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:56.439617 11783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "8c5f6d5680764192b47b819f664e78bf"
format_stamp: "Formatted at 2026-06-19 18:46:56 on dist-test-slave-mld5"
server_key: "da5eac35c6c55f6b6f2faddc4cb67434"
server_key_iv: "08c9a63e52c0b14f1c9832d5307773a8"
server_key_version: "encryptionkey@0"
I20260619 18:46:56.440179 11783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "8c5f6d5680764192b47b819f664e78bf"
format_stamp: "Formatted at 2026-06-19 18:46:56 on dist-test-slave-mld5"
server_key: "da5eac35c6c55f6b6f2faddc4cb67434"
server_key_iv: "08c9a63e52c0b14f1c9832d5307773a8"
server_key_version: "encryptionkey@0"
I20260619 18:46:56.443828 11783 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 18:46:56.446302 11798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:56.447662 11783 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:46:56.447801 11783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "8c5f6d5680764192b47b819f664e78bf"
format_stamp: "Formatted at 2026-06-19 18:46:56 on dist-test-slave-mld5"
server_key: "da5eac35c6c55f6b6f2faddc4cb67434"
server_key_iv: "08c9a63e52c0b14f1c9832d5307773a8"
server_key_version: "encryptionkey@0"
I20260619 18:46:56.447912 11783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:56.452423 11783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:56.455482 11783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:56.455677 11783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:56.462460 11783 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:39063
I20260619 18:46:56.462540 11850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:39063 every 8 connection(s)
I20260619 18:46:56.463521 11783 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:56.465809 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 11783
I20260619 18:46:56.465972 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:46:56.466261 22077 external_mini_cluster.cc:1428] Setting key f074861fecef7541450587f6669c5e1e
I20260619 18:46:56.466487 11851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:56.471529 11854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:56.467646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50395 (local address 127.21.143.126:39063)
0619 18:46:56.468093 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:56.468111 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:56.468149 (+ 38us) server_negotiation.cc:407] Connection header received
0619 18:46:56.468794 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:56.468810 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:56.469090 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:56.469474 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:56.470269 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:56.470274 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:56.470293 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:56.470304 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:56.470358 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:56.470378 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:56.470528 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:56.470598 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 18:46:56.470762 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":151,"threads_started":1}
I20260619 18:46:56.473368 11851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c5f6d5680764192b47b819f664e78bf: Bootstrap starting.
I20260619 18:46:56.474346 22077 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:56.475947 11851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c5f6d5680764192b47b819f664e78bf: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:56.476867 11851 log.cc:824] T 00000000000000000000000000000000 P 8c5f6d5680764192b47b819f664e78bf: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:56.478066 11854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:56.476404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55163 (local address 127.21.143.126:39063)
0619 18:46:56.476534 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:56.476538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:56.476683 (+ 145us) server_negotiation.cc:407] Connection header received
0619 18:46:56.476767 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:56.476770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:56.476804 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:56.476889 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:56.477660 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:56.477663 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:56.477666 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:56.477677 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:56.477707 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:56.477717 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:56.477804 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:56.477906 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 18:46:56.477949 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 18:46:56.478900 11851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c5f6d5680764192b47b819f664e78bf: No bootstrap required, opened a new log
I20260619 18:46:56.482573 11851 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c5f6d5680764192b47b819f664e78bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c5f6d5680764192b47b819f664e78bf" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39063 } }
I20260619 18:46:56.482882 11851 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c5f6d5680764192b47b819f664e78bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:56.482949 11851 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c5f6d5680764192b47b819f664e78bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c5f6d5680764192b47b819f664e78bf, State: Initialized, Role: FOLLOWER
I20260619 18:46:56.483465 11851 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c5f6d5680764192b47b819f664e78bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c5f6d5680764192b47b819f664e78bf" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39063 } }
I20260619 18:46:56.483605 11851 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c5f6d5680764192b47b819f664e78bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:56.483671 11851 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c5f6d5680764192b47b819f664e78bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:56.483762 11851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c5f6d5680764192b47b819f664e78bf [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:56.484616 11851 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c5f6d5680764192b47b819f664e78bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c5f6d5680764192b47b819f664e78bf" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39063 } }
I20260619 18:46:56.484989 11851 leader_election.cc:302] T 00000000000000000000000000000000 P 8c5f6d5680764192b47b819f664e78bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c5f6d5680764192b47b819f664e78bf; no voters:
I20260619 18:46:56.485306 11851 leader_election.cc:288] T 00000000000000000000000000000000 P 8c5f6d5680764192b47b819f664e78bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:56.485430 11861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c5f6d5680764192b47b819f664e78bf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:56.485695 11861 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c5f6d5680764192b47b819f664e78bf [term 1 LEADER]: Becoming Leader. State: Replica: 8c5f6d5680764192b47b819f664e78bf, State: Running, Role: LEADER
I20260619 18:46:56.486014 11861 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c5f6d5680764192b47b819f664e78bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c5f6d5680764192b47b819f664e78bf" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39063 } }
I20260619 18:46:56.486459 11851 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c5f6d5680764192b47b819f664e78bf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:56.487610 11863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c5f6d5680764192b47b819f664e78bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c5f6d5680764192b47b819f664e78bf. Latest consensus state: current_term: 1 leader_uuid: "8c5f6d5680764192b47b819f664e78bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c5f6d5680764192b47b819f664e78bf" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39063 } } }
I20260619 18:46:56.487754 11863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c5f6d5680764192b47b819f664e78bf [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:56.488125 11870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:56.488107 11862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c5f6d5680764192b47b819f664e78bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c5f6d5680764192b47b819f664e78bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c5f6d5680764192b47b819f664e78bf" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 39063 } } }
I20260619 18:46:56.488293 11862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c5f6d5680764192b47b819f664e78bf [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:56.491218 11870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:56.496479 11870 catalog_manager.cc:1347] Generated new cluster ID: 3d353cf857144c24a87affcc10c93dcc
I20260619 18:46:56.496562 11870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:56.511447 11870 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:56.511786 11870 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:56.518621 11870 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c5f6d5680764192b47b819f664e78bf: Generated new TSK 0
I20260619 18:46:56.519438 11870 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:56.524322 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 11783
2026-06-19T18:46:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1374 ms)
[----------] 18 tests from AuthTokenIssuingTest (25074 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:56 dist-test-slave-mld5 krb5kdc[11882](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:56 dist-test-slave-mld5 krb5kdc[11882](info): set up 2 sockets
Jun 19 18:46:56 dist-test-slave-mld5 krb5kdc[11882](info): commencing operation
krb5kdc: starting...
W20260619 18:46:57.571650 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:46:57 dist-test-slave-mld5 krb5kdc[11882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.143.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
I20260619 18:46:57.692013 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:43503
--webserver_interface=127.21.143.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:45619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:43503,127.21.143.125:39849,127.21.143.124:35573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:57.803704 11895 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 18:46:57.803983 11895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:57.804008 11895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:57.804085 11895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:57.808145 11895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:57.808197 11895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:57.808215 11895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:57.808231 11895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:57.808246 11895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:57.813400 11895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:45619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:43503,127.21.143.125:39849,127.21.143.124:35573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:43503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:57.814433 11895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:57.815521 11895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:57.823474 11903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:57.823459 11901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:57.823459 11900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:57.823726 11895 server_base.cc:1034] running on GCE node
I20260619 18:46:57.824321 11895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:57.825379 11895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:57.826568 11895 hybrid_clock.cc:648] HybridClock initialized: now 1781894817826541 us; error 42 us; skew 500 ppm
Jun 19 18:46:57 dist-test-slave-mld5 krb5kdc[11882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894817, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:57.829310 11895 init.cc:377] Logged in from keytab as kudu/127.21.143.126@KRBTEST.COM (short username kudu)
I20260619 18:46:57.830137 11895 webserver.cc:460] Webserver started at http://127.21.143.126:42411/ using document root <none> and password file <none>
I20260619 18:46:57.830693 11895 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:57.830780 11895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:57.831024 11895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:57.833110 11895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "85e4498da3794134a0c193468b1e9732"
format_stamp: "Formatted at 2026-06-19 18:46:57 on dist-test-slave-mld5"
server_key: "d50571a705392717ab2d18fcfcb223b9"
server_key_iv: "4291e72b1e41f9ce4f90af2e93764ba9"
server_key_version: "encryptionkey@0"
I20260619 18:46:57.833693 11895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "85e4498da3794134a0c193468b1e9732"
format_stamp: "Formatted at 2026-06-19 18:46:57 on dist-test-slave-mld5"
server_key: "d50571a705392717ab2d18fcfcb223b9"
server_key_iv: "4291e72b1e41f9ce4f90af2e93764ba9"
server_key_version: "encryptionkey@0"
I20260619 18:46:57.837572 11895 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260619 18:46:57.840037 11910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:57.841264 11895 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:46:57.841404 11895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "85e4498da3794134a0c193468b1e9732"
format_stamp: "Formatted at 2026-06-19 18:46:57 on dist-test-slave-mld5"
server_key: "d50571a705392717ab2d18fcfcb223b9"
server_key_iv: "4291e72b1e41f9ce4f90af2e93764ba9"
server_key_version: "encryptionkey@0"
I20260619 18:46:57.841517 11895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:57.857908 11895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:57.860527 11895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:57.860705 11895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:57.867069 11895 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:43503
I20260619 18:46:57.867400 11962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:43503 every 8 connection(s)
I20260619 18:46:57.868045 11895 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:57.868430 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 11895
I20260619 18:46:57.868572 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:46:57.868860 22077 external_mini_cluster.cc:1428] Setting key ff2f5b8d2f130d3d810732d6d6980993
I20260619 18:46:57.870786 11963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:57.874364 11963 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.143.126" port: 43503 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.143.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.125@KRBTEST.COM" created.
Jun 19 18:46:57 dist-test-slave-mld5 krb5kdc[11882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894817, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:46:57.890269 11967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:57.877165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53592 (local address 127.21.143.126:43503)
0619 18:46:57.877430 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:57.877438 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:57.877471 (+ 33us) server_negotiation.cc:407] Connection header received
0619 18:46:57.878566 (+ 1095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:57.878570 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:57.878735 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:57.878876 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:57.879941 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.880808 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.881853 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.882211 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.882324 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:57.885246 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:57.885252 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:57.885261 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:57.885282 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:57.887286 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:57.887794 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:57.887800 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:57.887802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:57.887986 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:57.888301 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:57.888308 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:57.888312 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:57.888870 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:57.889028 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:57.889373 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 18:46:57.889592 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":165,"thread_start_us":105,"threads_started":1}
I20260619 18:46:57.890268 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:57.876491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:43503 (local address 127.0.0.1:53592)
0619 18:46:57.876998 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:57.877063 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:57.878271 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:57.878938 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:57.878944 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:57.879402 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:57.879785 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:57.879805 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.880921 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.880924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:57.881750 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:57.881759 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.882389 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.882392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:57.882511 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:57.883101 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:57.883143 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:57.885093 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:57.887434 (+ 2341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:57.887447 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:57.887460 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:57.887678 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:57.888090 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:57.888092 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:57.888094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:57.888200 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:57.888985 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:57.888990 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:57.889106 (+ 116us) client_negotiation.cc:769] Sending connection context
0619 18:46:57.889251 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 18:46:57.889488 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":98,"threads_started":1}
I20260619 18:46:57.892863 11963 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39849 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:46:57.893610 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:57.893356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.125:39849 (local address 127.0.0.1:35104)
0619 18:46:57.893474 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:57.893553 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.125:39849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 18:46:57.893791 11913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.143.125:39849: connect: Connection refused (error 111)
I20260619 18:46:57.894569 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:57.894368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.125:39849 (local address 127.0.0.1:35116)
0619 18:46:57.894471 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:57.894522 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.125:39849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 18:46:57.894765 11963 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.143.125:39849: Network error: Client connection negotiation failed: client connection to 127.21.143.125:39849: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.125.keytab.
Entry for principal kudu/127.21.143.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.125.keytab.
I20260619 18:46:57.899067 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.125:39849
--webserver_interface=127.21.143.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.125.keytab
--principal=kudu/127.21.143.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:45619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:43503,127.21.143.125:39849,127.21.143.124:35573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:46:57.944008 11963 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39849 } attempt: 1
I20260619 18:46:57.944923 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:57.944607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.125:39849 (local address 127.0.0.1:35124)
0619 18:46:57.944770 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:57.944844 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.125:39849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 18:46:57.945670 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:57.945432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.125:39849 (local address 127.0.0.1:35140)
0619 18:46:57.945549 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:57.945606 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.125:39849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 18:46:57.945784 11963 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.143.125:39849: Network error: Client connection negotiation failed: client connection to 127.21.143.125:39849: connect: Connection refused (error 111)
W20260619 18:46:58.009629 11970 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 18:46:58.009967 11970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:58.010015 11970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:58.010089 11970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 18:46:58.013955 11963 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39849 } attempt: 2
W20260619 18:46:58.014056 11970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:58.014137 11970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:58.014183 11970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:58.014221 11970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:58.014256 11970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:58.014765 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.014477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.125:39849 (local address 127.0.0.1:35152)
0619 18:46:58.014617 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:58.014683 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.125:39849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 18:46:58.015440 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.015208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.125:39849 (local address 127.0.0.1:35154)
0619 18:46:58.015337 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:58.015382 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.125:39849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 18:46:58.015580 11963 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.143.125:39849: Network error: Client connection negotiation failed: client connection to 127.21.143.125:39849: connect: Connection refused (error 111)
I20260619 18:46:58.019680 11970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:45619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:43503,127.21.143.125:39849,127.21.143.124:35573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.125.keytab
--principal=kudu/127.21.143.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.125:39849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.143.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:58.020780 11970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:58.021812 11970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:58.030100 11975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:58.030128 11978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:58.030109 11976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:58.030531 11970 server_base.cc:1034] running on GCE node
I20260619 18:46:58.031139 11970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:58.032328 11970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:58.033532 11970 hybrid_clock.cc:648] HybridClock initialized: now 1781894818033515 us; error 28 us; skew 500 ppm
Jun 19 18:46:58 dist-test-slave-mld5 krb5kdc[11882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894818, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:58.036367 11970 init.cc:377] Logged in from keytab as kudu/127.21.143.125@KRBTEST.COM (short username kudu)
I20260619 18:46:58.037267 11970 webserver.cc:460] Webserver started at http://127.21.143.125:37869/ using document root <none> and password file <none>
I20260619 18:46:58.037842 11970 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:58.037948 11970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:58.038180 11970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:58.040272 11970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-1/data/instance:
uuid: "6ac6dac6124d4d7faf6996332f815f32"
format_stamp: "Formatted at 2026-06-19 18:46:58 on dist-test-slave-mld5"
server_key: "08b777bd6b309294e739edfaf7c3abc3"
server_key_iv: "aefa776ebb2c6ff8bcacc6cb31c7e714"
server_key_version: "encryptionkey@0"
I20260619 18:46:58.040814 11970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-1/wal/instance:
uuid: "6ac6dac6124d4d7faf6996332f815f32"
format_stamp: "Formatted at 2026-06-19 18:46:58 on dist-test-slave-mld5"
server_key: "08b777bd6b309294e739edfaf7c3abc3"
server_key_iv: "aefa776ebb2c6ff8bcacc6cb31c7e714"
server_key_version: "encryptionkey@0"
I20260619 18:46:58.044473 11970 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 18:46:58.046828 11985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:58.048065 11970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 18:46:58.048200 11970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-1/wal
uuid: "6ac6dac6124d4d7faf6996332f815f32"
format_stamp: "Formatted at 2026-06-19 18:46:58 on dist-test-slave-mld5"
server_key: "08b777bd6b309294e739edfaf7c3abc3"
server_key_iv: "aefa776ebb2c6ff8bcacc6cb31c7e714"
server_key_version: "encryptionkey@0"
I20260619 18:46:58.048308 11970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:58.062717 11970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:58.065641 11970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:58.065850 11970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:58.073092 11970 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.125:39849
I20260619 18:46:58.073361 12037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.125:39849 every 8 connection(s)
I20260619 18:46:58.074110 11970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-1/data/info.pb
I20260619 18:46:58.074970 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 11970
I20260619 18:46:58.075098 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-1/wal/instance
I20260619 18:46:58.075435 22077 external_mini_cluster.cc:1428] Setting key 229d5d97411ab8becd13c7d0dde981e9
I20260619 18:46:58.077519 12038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:58.081346 12038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.143.126" port: 43503 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.143.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.124@KRBTEST.COM" created.
Jun 19 18:46:58 dist-test-slave-mld5 krb5kdc[11882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894818, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.125@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:46:58.095579 11967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.083273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53598 (local address 127.21.143.126:43503)
0619 18:46:58.083452 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:58.083457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:58.084375 (+ 918us) server_negotiation.cc:407] Connection header received
0619 18:46:58.085381 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:58.085384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:58.085446 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:58.085551 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:58.086692 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.087266 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.088618 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.088883 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.088998 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:58.091336 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:58.091341 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:58.091343 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:58.091373 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:58.093272 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:58.093857 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:58.093860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:58.093861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:58.093934 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:58.094315 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:58.094320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:58.094322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:58.094499 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:58.094603 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:58.095210 (+ 607us) server_negotiation.cc:299] Negotiation successful
0619 18:46:58.095380 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 18:46:58.096398 12041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.083753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:43503 (local address 127.0.0.1:53598)
0619 18:46:58.084228 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:58.084270 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:58.085139 (+ 869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:58.085722 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:58.085729 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:58.086045 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:58.086502 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:58.086514 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.087432 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.087439 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:58.088460 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:58.088470 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.089035 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.089039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:58.089201 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:58.089822 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:58.089836 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:58.091132 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:58.093392 (+ 2260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:58.093398 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:58.093411 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:58.093717 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:58.094027 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:58.094031 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:58.094033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:58.094214 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:58.094624 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:58.094630 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:58.094964 (+ 334us) client_negotiation.cc:769] Sending connection context
0619 18:46:58.095153 (+ 189us) client_negotiation.cc:240] Negotiation successful
0619 18:46:58.095483 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":108,"threads_started":1}
I20260619 18:46:58.097819 12038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39849 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.124.keytab.
Entry for principal kudu/127.21.143.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.124.keytab.
I20260619 18:46:58.099517 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.124:35573
--webserver_interface=127.21.143.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.124.keytab
--principal=kudu/127.21.143.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:45619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:43503,127.21.143.125:39849,127.21.143.124:35573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:46:58 dist-test-slave-mld5 krb5kdc[11882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894818, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.125@KRBTEST.COM for kudu/127.21.143.125@KRBTEST.COM
I20260619 18:46:58.106755 11963 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39849 } attempt: 3
I20260619 18:46:58.106977 12041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.098362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.125:39849 (local address 127.0.0.1:35158)
0619 18:46:58.098475 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:58.098486 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:58.098563 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:58.099208 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:58.099211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:58.099397 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:58.099465 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:58.099470 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.100466 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.100470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:58.100962 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:58.100969 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.101523 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.101545 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:58.101666 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:58.102149 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:58.102160 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:58.103601 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:58.105431 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:58.105437 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:58.105440 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:58.105632 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:58.105916 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:58.105921 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:58.105923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:58.106003 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:58.106455 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:58.106458 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:58.106539 (+ 81us) client_negotiation.cc:769] Sending connection context
0619 18:46:58.106664 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 18:46:58.106799 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":3}
I20260619 18:46:58.107518 12043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.098492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35158 (local address 127.21.143.125:39849)
0619 18:46:58.098897 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:58.098913 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:58.098930 (+ 17us) server_negotiation.cc:407] Connection header received
0619 18:46:58.098961 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:58.098965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:58.099096 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:58.099229 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:58.099568 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.100349 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.101065 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.101416 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.101544 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:58.103751 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:58.103759 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:58.103771 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:58.103790 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:58.105320 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:58.105741 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:58.105752 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:58.105753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:58.105824 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:58.106093 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:58.106096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:58.106098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:58.106344 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:58.106485 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:58.106782 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 18:46:58.106909 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":313,"thread_start_us":172,"threads_started":1}
I20260619 18:46:58.109449 12038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35573 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:46:58.110160 12041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.109924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:35573 (local address 127.0.0.1:37096)
0619 18:46:58.110026 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:58.110101 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 18:46:58.110829 11989 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35573: connect: Connection refused (error 111)
I20260619 18:46:58.112571 12041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.112347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:35573 (local address 127.0.0.1:37102)
0619 18:46:58.112462 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:58.112508 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 18:46:58.112731 12038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.143.124:35573: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35573: connect: Connection refused (error 111)
Jun 19 18:46:58 dist-test-slave-mld5 krb5kdc[11882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894817, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for kudu/127.21.143.125@KRBTEST.COM
I20260619 18:46:58.116415 12045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.107352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35162 (local address 127.21.143.125:39849)
0619 18:46:58.107656 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:58.107668 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:58.107683 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:46:58.107720 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:58.107722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:58.107771 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:58.107869 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:58.108343 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.109263 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.110619 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.111025 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.111149 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:58.113338 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:58.113341 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:58.113343 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:58.113363 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:58.114850 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:58.115346 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:58.115355 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:58.115357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:58.115428 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:58.115695 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:58.115697 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:58.115698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:58.115859 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:58.115934 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:58.116199 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 18:46:58.116286 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":103,"threads_started":1}
I20260619 18:46:58.116581 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.107257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.125:39849 (local address 127.0.0.1:35162)
0619 18:46:58.107445 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:58.107458 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:58.107572 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:58.107911 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:58.107915 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:58.108085 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:58.108214 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:58.108220 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.109432 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.109438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:58.110485 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:58.110497 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.111144 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.111148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:58.111259 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:58.111775 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:58.111813 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:58.113203 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:58.114955 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:58.114960 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:58.114963 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:58.115217 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:58.115515 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:58.115518 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:58.115520 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:58.115610 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:58.115947 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:58.115950 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:58.116053 (+ 103us) client_negotiation.cc:769] Sending connection context
0619 18:46:58.116172 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 18:46:58.116330 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260619 18:46:58.117412 11963 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35573 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:46:58.118139 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.117906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:35573 (local address 127.0.0.1:37112)
0619 18:46:58.118021 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:58.118076 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 18:46:58.118806 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.118564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:35573 (local address 127.0.0.1:37114)
0619 18:46:58.118705 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:58.118753 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260619 18:46:58.118973 11963 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.143.124:35573: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35573: connect: Connection refused (error 111)
I20260619 18:46:58.150172 11963 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35573 } attempt: 1
I20260619 18:46:58.151084 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.150739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:35573 (local address 127.0.0.1:37130)
0619 18:46:58.150912 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:58.150990 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 18:46:58.151894 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.151646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:35573 (local address 127.0.0.1:37136)
0619 18:46:58.151774 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:58.151827 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 18:46:58.152046 11963 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.143.124:35573: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35573: connect: Connection refused (error 111)
I20260619 18:46:58.162000 12038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35573 } attempt: 1
I20260619 18:46:58.162820 12041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.162496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:35573 (local address 127.0.0.1:37152)
0619 18:46:58.162681 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:58.162748 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260619 18:46:58.163666 12041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.163420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:35573 (local address 127.0.0.1:37156)
0619 18:46:58.163523 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:58.163577 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 18:46:58.163800 12038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.143.124:35573: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35573: connect: Connection refused (error 111)
W20260619 18:46:58.214468 12044 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 18:46:58.214761 12044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:58.214828 12044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:58.214906 12044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:58.218890 12044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:58.218946 12044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:58.218966 12044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:58.219027 12044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:58.219053 12044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:58.224236 12044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:45619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:43503,127.21.143.125:39849,127.21.143.124:35573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.124.keytab
--principal=kudu/127.21.143.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.124:35573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.143.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:46:58.225291 12044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:58.226337 12044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 18:46:58.227218 11963 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35573 } attempt: 2
I20260619 18:46:58.228075 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.227790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:35573 (local address 127.0.0.1:37172)
0619 18:46:58.227942 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:58.228001 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 18:46:58.228808 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.228567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:35573 (local address 127.0.0.1:37188)
0619 18:46:58.228703 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:58.228750 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 18:46:58.228932 11963 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.143.124:35573: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35573: connect: Connection refused (error 111)
I20260619 18:46:58.231950 12038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35573 } attempt: 2
I20260619 18:46:58.232620 12041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.232357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:35573 (local address 127.0.0.1:37200)
0619 18:46:58.232483 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:58.232540 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 18:46:58.233264 12041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.233055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:35573 (local address 127.0.0.1:37212)
0619 18:46:58.233156 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:58.233208 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 18:46:58.233441 12038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.143.124:35573: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35573: connect: Connection refused (error 111)
W20260619 18:46:58.235383 12052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:58.235389 12054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:58.235636 12044 server_base.cc:1034] running on GCE node
W20260619 18:46:58.235450 12051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:58.236362 12044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:58.237475 12044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:58.238672 12044 hybrid_clock.cc:648] HybridClock initialized: now 1781894818238656 us; error 26 us; skew 500 ppm
Jun 19 18:46:58 dist-test-slave-mld5 krb5kdc[11882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894818, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:58.241353 12044 init.cc:377] Logged in from keytab as kudu/127.21.143.124@KRBTEST.COM (short username kudu)
I20260619 18:46:58.242295 12044 webserver.cc:460] Webserver started at http://127.21.143.124:40575/ using document root <none> and password file <none>
I20260619 18:46:58.242895 12044 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:58.242956 12044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:58.243187 12044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:58.245311 12044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-2/data/instance:
uuid: "a054b40ca18f4d52a1ab19e2adcea017"
format_stamp: "Formatted at 2026-06-19 18:46:58 on dist-test-slave-mld5"
server_key: "58f5a664144da5cf759d765614b6a900"
server_key_iv: "f493ccd2234458612c58b878a47a9db0"
server_key_version: "encryptionkey@0"
I20260619 18:46:58.245901 12044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-2/wal/instance:
uuid: "a054b40ca18f4d52a1ab19e2adcea017"
format_stamp: "Formatted at 2026-06-19 18:46:58 on dist-test-slave-mld5"
server_key: "58f5a664144da5cf759d765614b6a900"
server_key_iv: "f493ccd2234458612c58b878a47a9db0"
server_key_version: "encryptionkey@0"
I20260619 18:46:58.249640 12044 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:46:58.252064 12061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:58.253268 12044 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 18:46:58.253396 12044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-2/wal
uuid: "a054b40ca18f4d52a1ab19e2adcea017"
format_stamp: "Formatted at 2026-06-19 18:46:58 on dist-test-slave-mld5"
server_key: "58f5a664144da5cf759d765614b6a900"
server_key_iv: "f493ccd2234458612c58b878a47a9db0"
server_key_version: "encryptionkey@0"
I20260619 18:46:58.253516 12044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:58.267161 12044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:58.269927 12044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:58.270134 12044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:58.277228 12044 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.124:35573
I20260619 18:46:58.277522 12113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.124:35573 every 8 connection(s)
I20260619 18:46:58.278196 12044 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-2/data/info.pb
I20260619 18:46:58.280880 12114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:58.284431 12114 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.143.126" port: 43503 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:46:58.285621 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 12044
I20260619 18:46:58.285774 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894542572984-22077-0/minicluster-data/master-2/wal/instance
I20260619 18:46:58.286062 22077 external_mini_cluster.cc:1428] Setting key 72df8c4e3e678fe55fb75c7c3e9c832a
Jun 19 18:46:58 dist-test-slave-mld5 krb5kdc[11882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894818, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.124@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
Jun 19 18:46:58 dist-test-slave-mld5 krb5kdc[11882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.124@KRBTEST.COM
I20260619 18:46:58.300050 11967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.286352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53606 (local address 127.21.143.126:43503)
0619 18:46:58.286510 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:58.286515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:58.287571 (+ 1056us) server_negotiation.cc:407] Connection header received
0619 18:46:58.288318 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:58.288321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:58.288372 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:58.288470 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:58.289593 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.290373 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.291792 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.292136 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.292245 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:58.295325 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:58.295332 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:58.295338 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:58.295368 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:58.297070 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:58.298402 (+ 1332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:58.298406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:58.298407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:58.298521 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:58.298927 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:58.298933 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:58.298937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:58.299150 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:58.299249 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:58.299732 (+ 483us) server_negotiation.cc:299] Negotiation successful
0619 18:46:58.299877 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 18:46:58.300969 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.286788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37214 (local address 127.21.143.124:35573)
0619 18:46:58.287255 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:58.287277 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:58.287346 (+ 69us) server_negotiation.cc:407] Connection header received
0619 18:46:58.288094 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:58.288113 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:58.288354 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:58.288527 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:58.289449 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.290650 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.291794 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.292391 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.292486 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:58.296065 (+ 3579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:58.296071 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:58.296083 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:58.296105 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:58.297827 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:58.298404 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:58.298409 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:58.298411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:58.298558 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:58.298866 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:58.298868 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:58.298869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:58.299436 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:58.299580 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:58.299892 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 18:46:58.300120 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":114,"server-negotiator.queue_time_us":322,"spinlock_wait_cycles":9984,"thread_start_us":142,"threads_started":1}
I20260619 18:46:58.300983 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.286788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:43503 (local address 127.0.0.1:53606)
0619 18:46:58.287413 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:58.287461 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:58.288109 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:58.288554 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:58.288559 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:58.289063 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:58.289445 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:58.289465 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.290483 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.290486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:58.291655 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:58.291666 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.292283 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.292286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:58.292425 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:58.293270 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:58.293292 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:58.295146 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:58.297979 (+ 2833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:58.297988 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:58.298001 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:58.298297 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:58.298646 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:58.298650 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:58.298652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:58.298826 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:58.299255 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:58.299261 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:58.299450 (+ 189us) client_negotiation.cc:769] Sending connection context
0619 18:46:58.299664 (+ 214us) client_negotiation.cc:240] Negotiation successful
0619 18:46:58.300048 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":42,"spinlock_wait_cycles":53120,"thread_start_us":142,"threads_started":1}
I20260619 18:46:58.302323 12114 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39849 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:46:58 dist-test-slave-mld5 krb5kdc[11882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894818, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.124@KRBTEST.COM for kudu/127.21.143.125@KRBTEST.COM
Jun 19 18:46:58 dist-test-slave-mld5 krb5kdc[11882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:46:58.311889 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.302886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.125:39849 (local address 127.0.0.1:35170)
0619 18:46:58.303277 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:58.303291 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:58.303392 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:58.303616 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:58.303619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:58.303803 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:58.303938 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:58.303948 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.304971 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.304973 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:58.305822 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:58.305834 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.306398 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.306419 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:58.306522 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:58.307064 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:58.307082 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:58.308420 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:58.310332 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:58.310338 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:58.310341 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:58.310575 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:58.310849 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:58.310851 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:58.310852 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:58.310923 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:58.311383 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:58.311386 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:58.311481 (+ 95us) client_negotiation.cc:769] Sending connection context
0619 18:46:58.311590 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 18:46:58.311723 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194}
I20260619 18:46:58.311960 12045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.303006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35170 (local address 127.21.143.125:39849)
0619 18:46:58.303149 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:58.303153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:58.303367 (+ 214us) server_negotiation.cc:407] Connection header received
0619 18:46:58.303471 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:58.303473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:58.303518 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:58.303594 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:58.304081 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.304863 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.305940 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.306290 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.306403 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:58.308562 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:58.308568 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:58.308572 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:58.308602 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:58.310223 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:58.310670 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:58.310673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:58.310675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:58.310756 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:58.311044 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:58.311046 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:58.311047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:58.311277 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:58.311388 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:58.311651 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 18:46:58.311788 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 18:46:58.312251 11967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.303204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53608 (local address 127.21.143.126:43503)
0619 18:46:58.303348 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:58.303354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:58.303368 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:46:58.303468 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:58.303471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:58.303515 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:58.303594 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:58.304291 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.304995 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.306049 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.306363 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.306449 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:58.309063 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:58.309067 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:58.309069 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:58.309093 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:58.310519 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:58.310998 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:58.311004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:58.311007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:58.311087 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:58.311373 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:58.311376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:58.311378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:58.311617 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:58.311818 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:58.311966 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 18:46:58.312100 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 18:46:58.313128 12114 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35573 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:46:58 dist-test-slave-mld5 krb5kdc[11882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894818, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.124@KRBTEST.COM for kudu/127.21.143.124@KRBTEST.COM
Jun 19 18:46:58 dist-test-slave-mld5 krb5kdc[11882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.125@KRBTEST.COM
I20260619 18:46:58.320945 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.313608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:35573 (local address 127.0.0.1:37224)
0619 18:46:58.313716 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:58.313729 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:58.313800 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:58.314034 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:58.314036 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:58.314190 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:58.314245 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:58.314249 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.314948 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.314951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:58.315526 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:58.315534 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.315995 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.315998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:58.316070 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:58.316536 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:58.316552 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:58.317764 (+ 1212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:58.319561 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:58.319567 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:58.319569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:58.319747 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:58.320022 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:58.320029 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:58.320030 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:58.320137 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:58.320499 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:58.320501 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:58.320596 (+ 95us) client_negotiation.cc:769] Sending connection context
0619 18:46:58.320703 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 18:46:58.320844 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 18:46:58.321101 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.313757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37224 (local address 127.21.143.124:35573)
0619 18:46:58.313857 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:58.313863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:58.313875 (+ 12us) server_negotiation.cc:407] Connection header received
0619 18:46:58.313900 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:58.313903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:58.313941 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:58.314024 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:58.314352 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.314840 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.315632 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.315877 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.315992 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:58.317892 (+ 1900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:58.317896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:58.317898 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:58.317922 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:58.319464 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:58.319851 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:58.319857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:58.319858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:58.319930 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:58.320218 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:58.320221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:58.320222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:58.320405 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:58.320503 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:58.320775 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 18:46:58.320911 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":34}
I20260619 18:46:58.321802 12045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.313623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35178 (local address 127.21.143.125:39849)
0619 18:46:58.313782 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:58.313784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:58.313793 (+ 9us) server_negotiation.cc:407] Connection header received
0619 18:46:58.313820 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:58.313821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:58.313851 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:58.313937 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:58.314493 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.314956 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.315784 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.316012 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.316122 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:58.318443 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:58.318446 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:58.318449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:58.318478 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:58.319999 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:58.320413 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:58.320416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:58.320417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:58.320507 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:58.320791 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:58.320794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:58.320796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:58.320991 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:58.321209 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:58.321528 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 18:46:58.321648 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260619 18:46:58.324317 12114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a054b40ca18f4d52a1ab19e2adcea017: Bootstrap starting.
I20260619 18:46:58.324574 12038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35573 } attempt: 3
I20260619 18:46:58.325639 22077 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:58.327051 12114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a054b40ca18f4d52a1ab19e2adcea017: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:58.328003 12114 log.cc:824] T 00000000000000000000000000000000 P a054b40ca18f4d52a1ab19e2adcea017: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:58.326805 11937 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "85e4498da3794134a0c193468b1e9732"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53608
I20260619 18:46:58.328179 11963 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35573 } attempt: 3
Jun 19 18:46:58 dist-test-slave-mld5 krb5kdc[11882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894818, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.125@KRBTEST.COM for kudu/127.21.143.124@KRBTEST.COM
I20260619 18:46:58.330242 12114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a054b40ca18f4d52a1ab19e2adcea017: No bootstrap required, opened a new log
I20260619 18:46:58.333853 12041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.324947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:35573 (local address 127.0.0.1:37228)
0619 18:46:58.325048 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:58.325062 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:58.325141 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:58.325429 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:58.325431 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:58.325608 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:58.325705 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:58.325712 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.326621 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.326624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:58.327421 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:58.327431 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.328005 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.328009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:58.328089 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:58.328974 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:58.328987 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:58.330403 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:58.332249 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:58.332253 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:58.332257 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:58.332512 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:58.332798 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:58.332804 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:58.332806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:58.332892 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:58.333332 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:58.333335 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:58.333435 (+ 100us) client_negotiation.cc:769] Sending connection context
0619 18:46:58.333556 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 18:46:58.333685 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 18:46:58.334353 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.325117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37228 (local address 127.21.143.124:35573)
0619 18:46:58.325227 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:58.325232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:58.325245 (+ 13us) server_negotiation.cc:407] Connection header received
0619 18:46:58.325277 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:58.325279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:58.325315 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:58.325394 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:58.325820 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.326479 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.327548 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.327876 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.327996 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:58.330518 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:58.330522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:58.330525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:58.330552 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:58.332119 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:58.332615 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:58.332618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:58.332620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:58.332698 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:58.332991 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:58.332996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:58.332998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:58.333243 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:58.333348 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:58.334123 (+ 775us) server_negotiation.cc:299] Negotiation successful
0619 18:46:58.334229 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 18:46:58 dist-test-slave-mld5 krb5kdc[11882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894817, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for kudu/127.21.143.124@KRBTEST.COM
I20260619 18:46:58.334622 12114 raft_consensus.cc:348] T 00000000000000000000000000000000 P a054b40ca18f4d52a1ab19e2adcea017 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e4498da3794134a0c193468b1e9732" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 43503 } } peers { permanent_uuid: "6ac6dac6124d4d7faf6996332f815f32" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39849 } } peers { permanent_uuid: "a054b40ca18f4d52a1ab19e2adcea017" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35573 } }
I20260619 18:46:58.334906 12114 raft_consensus.cc:729] T 00000000000000000000000000000000 P a054b40ca18f4d52a1ab19e2adcea017 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a054b40ca18f4d52a1ab19e2adcea017, State: Initialized, Role: FOLLOWER
I20260619 18:46:58.335255 12114 consensus_queue.cc:260] T 00000000000000000000000000000000 P a054b40ca18f4d52a1ab19e2adcea017 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e4498da3794134a0c193468b1e9732" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 43503 } } peers { permanent_uuid: "6ac6dac6124d4d7faf6996332f815f32" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39849 } } peers { permanent_uuid: "a054b40ca18f4d52a1ab19e2adcea017" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35573 } }
I20260619 18:46:58.335927 12121 sys_catalog.cc:455] T 00000000000000000000000000000000 P a054b40ca18f4d52a1ab19e2adcea017 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e4498da3794134a0c193468b1e9732" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 43503 } } peers { permanent_uuid: "6ac6dac6124d4d7faf6996332f815f32" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39849 } } peers { permanent_uuid: "a054b40ca18f4d52a1ab19e2adcea017" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35573 } } }
I20260619 18:46:58.336215 12121 sys_catalog.cc:458] T 00000000000000000000000000000000 P a054b40ca18f4d52a1ab19e2adcea017 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:46:58.336851 12114 sys_catalog.cc:564] T 00000000000000000000000000000000 P a054b40ca18f4d52a1ab19e2adcea017 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:58.337762 12038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ac6dac6124d4d7faf6996332f815f32: Bootstrap starting.
I20260619 18:46:58.338776 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.328531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:35573 (local address 127.0.0.1:37230)
0619 18:46:58.328692 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:58.328706 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:58.328785 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:58.329481 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:58.329486 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:58.329670 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:58.329781 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:58.329787 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.331032 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.331036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:58.332145 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:58.332157 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.332752 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.332755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:58.332839 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:58.333397 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:58.333413 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:58.334996 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:58.337031 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:58.337034 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:58.337036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:58.337287 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:58.337626 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:58.337630 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:58.337632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:58.337755 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:58.338236 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:58.338239 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:58.338331 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 18:46:58.338416 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 18:46:58.338573 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 18:46:58.339499 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.328956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37230 (local address 127.21.143.124:35573)
0619 18:46:58.329242 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:58.329250 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:58.329264 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:46:58.329296 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:58.329298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:58.329334 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:58.329420 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:58.329910 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.330630 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.332298 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.332621 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.333212 (+ 591us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:58.335150 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:58.335154 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:58.335156 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:58.335179 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:58.336890 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:58.337420 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:58.337423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:58.337424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:58.337513 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:58.337888 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:58.337894 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:58.337896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:58.338119 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:58.339098 (+ 979us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:58.339174 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 18:46:58.339337 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":84,"threads_started":1}
I20260619 18:46:58.341012 12038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ac6dac6124d4d7faf6996332f815f32: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:58.341854 12038 log.cc:824] T 00000000000000000000000000000000 P 6ac6dac6124d4d7faf6996332f815f32: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:58.342617 11963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85e4498da3794134a0c193468b1e9732: Bootstrap starting.
W20260619 18:46:58.342800 12132 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a054b40ca18f4d52a1ab19e2adcea017: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:46:58.342873 12132 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:46:58.345402 11963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85e4498da3794134a0c193468b1e9732: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:58.345976 12038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ac6dac6124d4d7faf6996332f815f32: No bootstrap required, opened a new log
I20260619 18:46:58.346298 11963 log.cc:824] T 00000000000000000000000000000000 P 85e4498da3794134a0c193468b1e9732: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:58.343588 12044 master_runner.cc:186] Error getting master registration for 127.21.143.126:43503: OK, instance_id { permanent_uuid: "85e4498da3794134a0c193468b1e9732" instance_seqno: 1781894817865248 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 18:46:58.347012 12044 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 18:46:58.348413 11963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85e4498da3794134a0c193468b1e9732: No bootstrap required, opened a new log
I20260619 18:46:58.350762 12038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ac6dac6124d4d7faf6996332f815f32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e4498da3794134a0c193468b1e9732" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 43503 } } peers { permanent_uuid: "6ac6dac6124d4d7faf6996332f815f32" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39849 } } peers { permanent_uuid: "a054b40ca18f4d52a1ab19e2adcea017" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35573 } }
I20260619 18:46:58.350987 12038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ac6dac6124d4d7faf6996332f815f32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ac6dac6124d4d7faf6996332f815f32, State: Initialized, Role: FOLLOWER
I20260619 18:46:58.351423 11963 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85e4498da3794134a0c193468b1e9732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e4498da3794134a0c193468b1e9732" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 43503 } } peers { permanent_uuid: "6ac6dac6124d4d7faf6996332f815f32" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39849 } } peers { permanent_uuid: "a054b40ca18f4d52a1ab19e2adcea017" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35573 } }
I20260619 18:46:58.351656 11963 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85e4498da3794134a0c193468b1e9732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e4498da3794134a0c193468b1e9732, State: Initialized, Role: FOLLOWER
I20260619 18:46:58.351490 12038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ac6dac6124d4d7faf6996332f815f32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e4498da3794134a0c193468b1e9732" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 43503 } } peers { permanent_uuid: "6ac6dac6124d4d7faf6996332f815f32" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39849 } } peers { permanent_uuid: "a054b40ca18f4d52a1ab19e2adcea017" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35573 } }
I20260619 18:46:58.352030 11963 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85e4498da3794134a0c193468b1e9732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e4498da3794134a0c193468b1e9732" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 43503 } } peers { permanent_uuid: "6ac6dac6124d4d7faf6996332f815f32" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39849 } } peers { permanent_uuid: "a054b40ca18f4d52a1ab19e2adcea017" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35573 } }
I20260619 18:46:58.352253 12136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ac6dac6124d4d7faf6996332f815f32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e4498da3794134a0c193468b1e9732" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 43503 } } peers { permanent_uuid: "6ac6dac6124d4d7faf6996332f815f32" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39849 } } peers { permanent_uuid: "a054b40ca18f4d52a1ab19e2adcea017" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35573 } } }
I20260619 18:46:58.352586 12136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ac6dac6124d4d7faf6996332f815f32 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:46:58.353099 12038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ac6dac6124d4d7faf6996332f815f32 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:58.353255 11963 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85e4498da3794134a0c193468b1e9732 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:58.353327 11937 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85e4498da3794134a0c193468b1e9732 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 18:46:58.353462 11937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85e4498da3794134a0c193468b1e9732 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:58.354904 11937 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85e4498da3794134a0c193468b1e9732 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e4498da3794134a0c193468b1e9732" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 43503 } } peers { permanent_uuid: "6ac6dac6124d4d7faf6996332f815f32" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39849 } } peers { permanent_uuid: "a054b40ca18f4d52a1ab19e2adcea017" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35573 } }
I20260619 18:46:58.356559 12012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "85e4498da3794134a0c193468b1e9732" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6ac6dac6124d4d7faf6996332f815f32"
I20260619 18:46:58.356777 12012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ac6dac6124d4d7faf6996332f815f32 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:58.356722 12088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "85e4498da3794134a0c193468b1e9732" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a054b40ca18f4d52a1ab19e2adcea017"
I20260619 18:46:58.356961 12088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a054b40ca18f4d52a1ab19e2adcea017 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:58.358165 12012 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6ac6dac6124d4d7faf6996332f815f32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85e4498da3794134a0c193468b1e9732 in term 1.
I20260619 18:46:58.358240 12088 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a054b40ca18f4d52a1ab19e2adcea017 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85e4498da3794134a0c193468b1e9732 in term 1.
I20260619 18:46:58.356024 11937 leader_election.cc:288] T 00000000000000000000000000000000 P 85e4498da3794134a0c193468b1e9732 [CANDIDATE]: Term 1 election: Requested vote from peers 6ac6dac6124d4d7faf6996332f815f32 (127.21.143.125:39849), a054b40ca18f4d52a1ab19e2adcea017 (127.21.143.124:35573)
I20260619 18:46:58.359093 11914 leader_election.cc:302] T 00000000000000000000000000000000 P 85e4498da3794134a0c193468b1e9732 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e4498da3794134a0c193468b1e9732, a054b40ca18f4d52a1ab19e2adcea017; no voters:
I20260619 18:46:58.359705 12161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e4498da3794134a0c193468b1e9732 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e4498da3794134a0c193468b1e9732" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 43503 } } peers { permanent_uuid: "6ac6dac6124d4d7faf6996332f815f32" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39849 } } peers { permanent_uuid: "a054b40ca18f4d52a1ab19e2adcea017" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35573 } } }
W20260619 18:46:58.359879 12159 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 85e4498da3794134a0c193468b1e9732: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:46:58.359985 12159 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:46:58.360173 12137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85e4498da3794134a0c193468b1e9732 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:58.360344 12137 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85e4498da3794134a0c193468b1e9732 [term 1 LEADER]: Becoming Leader. State: Replica: 85e4498da3794134a0c193468b1e9732, State: Running, Role: LEADER
I20260619 18:46:58.359895 12161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e4498da3794134a0c193468b1e9732 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:46:58.360649 12137 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85e4498da3794134a0c193468b1e9732 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e4498da3794134a0c193468b1e9732" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 43503 } } peers { permanent_uuid: "6ac6dac6124d4d7faf6996332f815f32" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39849 } } peers { permanent_uuid: "a054b40ca18f4d52a1ab19e2adcea017" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35573 } }
W20260619 18:46:58.362072 11916 tablet.cc:2369] T 00000000000000000000000000000000 P 85e4498da3794134a0c193468b1e9732: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:46:58.362946 12166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e4498da3794134a0c193468b1e9732 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85e4498da3794134a0c193468b1e9732. Latest consensus state: current_term: 1 leader_uuid: "85e4498da3794134a0c193468b1e9732" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e4498da3794134a0c193468b1e9732" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 43503 } } peers { permanent_uuid: "6ac6dac6124d4d7faf6996332f815f32" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39849 } } peers { permanent_uuid: "a054b40ca18f4d52a1ab19e2adcea017" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35573 } } }
I20260619 18:46:58.363104 12166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e4498da3794134a0c193468b1e9732 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:58.365214 12168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:58.365762 11970 master_runner.cc:263] No leader master found from master 6ac6dac6124d4d7faf6996332f815f32
I20260619 18:46:58.365823 11970 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 18:46:58.365932 11895 master_runner.cc:263] No leader master found from master 85e4498da3794134a0c193468b1e9732
I20260619 18:46:58.365979 11895 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 18:46:58.366034 12168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260619 18:46:58.368760 12158 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6ac6dac6124d4d7faf6996332f815f32: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:46:58.368839 12158 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:46:58.376642 12087 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a054b40ca18f4d52a1ab19e2adcea017 [term 1 FOLLOWER]: Refusing update from remote peer 85e4498da3794134a0c193468b1e9732: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:46:58.377285 12166 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 85e4498da3794134a0c193468b1e9732 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a054b40ca18f4d52a1ab19e2adcea017" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:46:58.381047 12121 sys_catalog.cc:455] T 00000000000000000000000000000000 P a054b40ca18f4d52a1ab19e2adcea017 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85e4498da3794134a0c193468b1e9732. Latest consensus state: current_term: 1 leader_uuid: "85e4498da3794134a0c193468b1e9732" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e4498da3794134a0c193468b1e9732" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 43503 } } peers { permanent_uuid: "6ac6dac6124d4d7faf6996332f815f32" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39849 } } peers { permanent_uuid: "a054b40ca18f4d52a1ab19e2adcea017" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35573 } } }
I20260619 18:46:58.381170 12121 sys_catalog.cc:458] T 00000000000000000000000000000000 P a054b40ca18f4d52a1ab19e2adcea017 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:46:58.382130 12012 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6ac6dac6124d4d7faf6996332f815f32 [term 1 FOLLOWER]: Refusing update from remote peer 85e4498da3794134a0c193468b1e9732: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:46:58.382733 12166 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 85e4498da3794134a0c193468b1e9732 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ac6dac6124d4d7faf6996332f815f32" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:46:58.383474 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.364068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37240 (local address 127.21.143.124:35573)
0619 18:46:58.364181 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:58.364185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:58.366699 (+ 2514us) server_negotiation.cc:407] Connection header received
0619 18:46:58.366881 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:58.366884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:58.366928 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:58.366975 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:58.368025 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.368773 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.371395 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.371759 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.371837 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:58.379420 (+ 7583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:58.379423 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:58.379426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:58.379454 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:58.381186 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:58.381906 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:58.381909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:58.381911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:58.382002 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:58.382382 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:58.382386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:58.382387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:58.382622 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:58.382743 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:58.383176 (+ 433us) server_negotiation.cc:299] Negotiation successful
0619 18:46:58.383324 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 18:46:58.386329 12136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ac6dac6124d4d7faf6996332f815f32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85e4498da3794134a0c193468b1e9732. Latest consensus state: current_term: 1 leader_uuid: "85e4498da3794134a0c193468b1e9732" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e4498da3794134a0c193468b1e9732" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 43503 } } peers { permanent_uuid: "6ac6dac6124d4d7faf6996332f815f32" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39849 } } peers { permanent_uuid: "a054b40ca18f4d52a1ab19e2adcea017" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35573 } } }
I20260619 18:46:58.386446 12136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ac6dac6124d4d7faf6996332f815f32 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:46:58.386706 12045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.364642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35186 (local address 127.21.143.125:39849)
0619 18:46:58.364742 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:58.364746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:58.369283 (+ 4537us) server_negotiation.cc:407] Connection header received
0619 18:46:58.369451 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:58.369454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:58.369496 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:58.369544 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:58.370427 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.371175 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.372256 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.372570 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.372633 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:58.375388 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:58.375391 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:58.375394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:58.375419 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:58.377573 (+ 2154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:58.379391 (+ 1818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:58.379395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:58.379397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:58.379487 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:58.385194 (+ 5707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:58.385198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:58.385200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:58.385504 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:58.385623 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:58.386399 (+ 776us) server_negotiation.cc:299] Negotiation successful
0619 18:46:58.386523 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 18:46:58.389364 11967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.363658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53610 (local address 127.21.143.126:43503)
0619 18:46:58.363765 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:58.363770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:58.368109 (+ 4339us) server_negotiation.cc:407] Connection header received
0619 18:46:58.368261 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:58.368264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:58.368303 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:58.368348 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:58.370254 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.370960 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.373031 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.373363 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.373452 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:58.375180 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:58.375185 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:58.375188 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:58.375213 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:58.378605 (+ 3392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:58.385699 (+ 7094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:58.385703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:58.385705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:58.385797 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:58.387140 (+ 1343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:58.387144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:58.387146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:58.387405 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:58.387520 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:58.389070 (+ 1550us) server_negotiation.cc:299] Negotiation successful
0619 18:46:58.389217 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 18:46:58.392220 12137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e4498da3794134a0c193468b1e9732 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "85e4498da3794134a0c193468b1e9732" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e4498da3794134a0c193468b1e9732" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 43503 } } peers { permanent_uuid: "6ac6dac6124d4d7faf6996332f815f32" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39849 } } peers { permanent_uuid: "a054b40ca18f4d52a1ab19e2adcea017" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35573 } } }
I20260619 18:46:58.392336 12137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e4498da3794134a0c193468b1e9732 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:58.392426 12137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e4498da3794134a0c193468b1e9732 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "85e4498da3794134a0c193468b1e9732" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e4498da3794134a0c193468b1e9732" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 43503 } } peers { permanent_uuid: "6ac6dac6124d4d7faf6996332f815f32" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39849 } } peers { permanent_uuid: "a054b40ca18f4d52a1ab19e2adcea017" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35573 } } }
I20260619 18:46:58.392493 12137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e4498da3794134a0c193468b1e9732 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:58.393363 12136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ac6dac6124d4d7faf6996332f815f32 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "85e4498da3794134a0c193468b1e9732" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e4498da3794134a0c193468b1e9732" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 43503 } } peers { permanent_uuid: "6ac6dac6124d4d7faf6996332f815f32" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39849 } } peers { permanent_uuid: "a054b40ca18f4d52a1ab19e2adcea017" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35573 } } }
I20260619 18:46:58.393496 12136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ac6dac6124d4d7faf6996332f815f32 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:46:58.393940 12168 catalog_manager.cc:1347] Generated new cluster ID: daa3e8ef31e540ae8a8303d56c2311f7
I20260619 18:46:58.393994 12168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:58.400624 12121 sys_catalog.cc:455] T 00000000000000000000000000000000 P a054b40ca18f4d52a1ab19e2adcea017 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "85e4498da3794134a0c193468b1e9732" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e4498da3794134a0c193468b1e9732" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 43503 } } peers { permanent_uuid: "6ac6dac6124d4d7faf6996332f815f32" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 39849 } } peers { permanent_uuid: "a054b40ca18f4d52a1ab19e2adcea017" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35573 } } }
I20260619 18:46:58.400744 12121 sys_catalog.cc:458] T 00000000000000000000000000000000 P a054b40ca18f4d52a1ab19e2adcea017 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:46:58.401324 12173 mvcc.cc:205] Tried to move back new op lower bound from 7298641176062722048 to 7298641176009408512. Current Snapshot: MvccSnapshot[applied={T|T < 7298641176062722048 or (T in {7298641176062722048})}]
I20260619 18:46:58.410715 12174 mvcc.cc:205] Tried to move back new op lower bound from 7298641176062722048 to 7298641176009408512. Current Snapshot: MvccSnapshot[applied={T|T < 7298641176062722048}]
I20260619 18:46:58.412742 12176 mvcc.cc:205] Tried to move back new op lower bound from 7298641176062722048 to 7298641176009408512. Current Snapshot: MvccSnapshot[applied={T|T < 7298641176062722048}]
I20260619 18:46:58.458643 12168 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:58.462359 12168 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:58.471858 12168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85e4498da3794134a0c193468b1e9732: Generated new TSK 0
I20260619 18:46:58.472711 12168 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 18:46:58.505103 12120 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:58.501728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37254 (local address 127.21.143.124:35573)
0619 18:46:58.501884 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:58.501889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:58.503892 (+ 2003us) server_negotiation.cc:407] Connection header received
0619 18:46:58.504097 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:58.504101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:58.504155 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:58.504214 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:58.504938 (+ 724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37254: BlockingRecv error: recv got EOF from 127.0.0.1:37254 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 18:46:58.506937 11967 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:58.502565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53616 (local address 127.21.143.126:43503)
0619 18:46:58.502711 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:58.502715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:58.505526 (+ 2811us) server_negotiation.cc:407] Connection header received
0619 18:46:58.506135 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:58.506139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:58.506194 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:58.506247 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:58.506808 (+ 561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53616: BlockingRecv error: recv got EOF from 127.0.0.1:53616 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260619 18:46:58.507978 12045 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:58.503233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35196 (local address 127.21.143.125:39849)
0619 18:46:58.503371 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:58.503376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:58.505872 (+ 2496us) server_negotiation.cc:407] Connection header received
0619 18:46:58.507205 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:58.507208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:58.507260 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:58.507364 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:58.507867 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35196: BlockingRecv error: recv got EOF from 127.0.0.1:35196 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 18:46:58.512641 12045 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:58.511116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35210 (local address 127.21.143.125:39849)
0619 18:46:58.511220 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:58.511225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:58.511705 (+ 480us) server_negotiation.cc:407] Connection header received
0619 18:46:58.511910 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:58.511913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:58.511988 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:58.512046 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:58.512530 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35210: BlockingRecv error: recv got EOF from 127.0.0.1:35210 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 18:46:58.517819 12045 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:58.516243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35214 (local address 127.21.143.125:39849)
0619 18:46:58.516351 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:58.516355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:58.516889 (+ 534us) server_negotiation.cc:407] Connection header received
0619 18:46:58.517077 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:58.517080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:58.517133 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:58.517189 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:58.517723 (+ 534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35214: BlockingRecv error: recv got EOF from 127.0.0.1:35214 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 18:46:58.523540 12045 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:58.521836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35220 (local address 127.21.143.125:39849)
0619 18:46:58.521943 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:58.521947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:58.522486 (+ 539us) server_negotiation.cc:407] Connection header received
0619 18:46:58.522705 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:58.522708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:58.522760 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:58.522814 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:58.523440 (+ 626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35220: BlockingRecv error: recv got EOF from 127.0.0.1:35220 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260619 18:46:58.531091 12045 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:58.529482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35232 (local address 127.21.143.125:39849)
0619 18:46:58.529625 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:58.529629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:58.530157 (+ 528us) server_negotiation.cc:407] Connection header received
0619 18:46:58.530352 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:58.530355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:58.530406 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:58.530459 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:58.530994 (+ 535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35232: BlockingRecv error: recv got EOF from 127.0.0.1:35232 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260619 18:46:58.542786 12045 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:58.541215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35234 (local address 127.21.143.125:39849)
0619 18:46:58.541324 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:58.541328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:58.541852 (+ 524us) server_negotiation.cc:407] Connection header received
0619 18:46:58.542035 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:58.542039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:58.542087 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:58.542138 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:58.542680 (+ 542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35234: BlockingRecv error: recv got EOF from 127.0.0.1:35234 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 18:46:58.569252 12045 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:58.567732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35238 (local address 127.21.143.125:39849)
0619 18:46:58.567864 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:58.567869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:58.568334 (+ 465us) server_negotiation.cc:407] Connection header received
0619 18:46:58.568495 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:58.568499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:58.568555 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:58.568645 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:58.569126 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35238: BlockingRecv error: recv got EOF from 127.0.0.1:35238 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 18:46:58.604777 12045 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:58.603682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35240 (local address 127.21.143.125:39849)
0619 18:46:58.603835 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:58.603841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:58.603980 (+ 139us) server_negotiation.cc:407] Connection header received
0619 18:46:58.604086 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:58.604088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:58.604134 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:58.604202 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:58.604649 (+ 447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35240: BlockingRecv error: recv got EOF from 127.0.0.1:35240 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 18:46:58.672228 12045 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:58.670939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35250 (local address 127.21.143.125:39849)
0619 18:46:58.671116 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:58.671120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:58.671171 (+ 51us) server_negotiation.cc:407] Connection header received
0619 18:46:58.671294 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:58.671335 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:58.671397 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:58.671474 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:58.672097 (+ 623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35250: BlockingRecv error: recv got EOF from 127.0.0.1:35250 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260619 18:46:58.803450 12045 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:58.802210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35260 (local address 127.21.143.125:39849)
0619 18:46:58.802376 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:58.802380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:58.802508 (+ 128us) server_negotiation.cc:407] Connection header received
0619 18:46:58.802693 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:58.802697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:58.802753 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:58.802847 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:58.803330 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35260: BlockingRecv error: recv got EOF from 127.0.0.1:35260 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260619 18:46:59.062570 12045 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:59.061440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35268 (local address 127.21.143.125:39849)
0619 18:46:59.061582 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:59.061587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:59.061696 (+ 109us) server_negotiation.cc:407] Connection header received
0619 18:46:59.061855 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:59.061858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:59.061909 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:59.061991 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:59.062452 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35268: BlockingRecv error: recv got EOF from 127.0.0.1:35268 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 18:46:59.344197 12132 catalog_manager.cc:1259] Loaded cluster ID: daa3e8ef31e540ae8a8303d56c2311f7
I20260619 18:46:59.344285 12132 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a054b40ca18f4d52a1ab19e2adcea017: loading cluster ID for follower catalog manager: success
I20260619 18:46:59.346115 12132 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a054b40ca18f4d52a1ab19e2adcea017: acquiring CA information for follower catalog manager: success
I20260619 18:46:59.347759 12132 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a054b40ca18f4d52a1ab19e2adcea017: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 18:46:59.370465 12158 catalog_manager.cc:1259] Loaded cluster ID: daa3e8ef31e540ae8a8303d56c2311f7
I20260619 18:46:59.370543 12158 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6ac6dac6124d4d7faf6996332f815f32: loading cluster ID for follower catalog manager: success
I20260619 18:46:59.372529 12158 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6ac6dac6124d4d7faf6996332f815f32: acquiring CA information for follower catalog manager: success
I20260619 18:46:59.373829 12158 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6ac6dac6124d4d7faf6996332f815f32: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 18:46:59.580672 12245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:59.576623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35270 (local address 127.21.143.125:39849)
0619 18:46:59.576862 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:59.576866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:59.576955 (+ 89us) server_negotiation.cc:407] Connection header received
0619 18:46:59.577140 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:59.577144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:59.577280 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:59.577396 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:46:59.577671 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:59.578209 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:59.579083 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:59.579388 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:59.579500 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:59.579826 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:46:59.580271 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:46:59.580410 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:59.580451 (+ 41us) server_negotiation.cc:299] Negotiation successful
0619 18:46:59.580508 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":64,"threads_started":1}
I20260619 18:46:59.586830 12251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:59.583482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37266 (local address 127.21.143.124:35573)
0619 18:46:59.583754 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:59.583758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:59.583771 (+ 13us) server_negotiation.cc:407] Connection header received
0619 18:46:59.583880 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:59.583883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:59.583984 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:59.584090 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:46:59.584264 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:59.584775 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:59.585521 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:59.585743 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:59.585815 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:59.586055 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:46:59.586452 (+ 397us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:46:59.586544 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:59.586590 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 18:46:59.586633 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":108,"threads_started":1}
I20260619 18:46:59.594115 12251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:59.589909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37268 (local address 127.21.143.124:35573)
0619 18:46:59.590033 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:59.590037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:59.590270 (+ 233us) server_negotiation.cc:407] Connection header received
0619 18:46:59.590444 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:59.590449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:59.590492 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:59.590580 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:46:59.590904 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:59.591536 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:59.592662 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:59.593008 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:59.593069 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:59.593619 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:46:59.593709 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:46:59.593778 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:59.593910 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 18:46:59.593969 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 18:46:59.594165 12245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:59.590042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35276 (local address 127.21.143.125:39849)
0619 18:46:59.590167 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:59.590171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:59.590422 (+ 251us) server_negotiation.cc:407] Connection header received
0619 18:46:59.590599 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:59.590602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:59.590658 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:59.590728 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:46:59.590989 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:59.591799 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:59.592964 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:59.593298 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:59.593405 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:59.593716 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:46:59.593795 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:46:59.593868 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:59.593949 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 18:46:59.594002 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 18:46:59.594671 12258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:59.589927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53628 (local address 127.21.143.126:43503)
0619 18:46:59.590402 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:59.590408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:59.590423 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:46:59.590514 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:59.590520 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:59.590671 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:59.590831 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:46:59.591035 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:59.591845 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:59.592753 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:59.593067 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:59.593174 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:59.593700 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:46:59.594045 (+ 345us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:46:59.594404 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:59.594448 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 18:46:59.594513 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":70,"threads_started":1}
I20260619 18:46:59.597406 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 11895
I20260619 18:46:59.603987 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 11970
I20260619 18:46:59.610105 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 12044
2026-06-19T18:46:59Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3088 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:59 dist-test-slave-mld5 krb5kdc[12262](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:59 dist-test-slave-mld5 krb5kdc[12262](info): set up 2 sockets
Jun 19 18:46:59 dist-test-slave-mld5 krb5kdc[12262](info): commencing operation
krb5kdc: starting...
W20260619 18:47:00.659726 22077 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/krb5kdc/test-user.keytab.
Jun 19 18:47:00 dist-test-slave-mld5 krb5kdc[12262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:47:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:47:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.143.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
Entry for principal kudu/127.21.143.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab.
I20260619 18:47:00.778033 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:37735
--webserver_interface=127.21.143.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:39963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:37735,127.21.143.125:40255,127.21.143.124:35669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:00.889792 12275 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 18:47:00.890162 12275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:00.890241 12275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:00.890316 12275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:00.895190 12275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:47:00.895267 12275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:00.895304 12275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:00.895358 12275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:47:00.895394 12275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:47:00.900437 12275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:39963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:37735,127.21.143.125:40255,127.21.143.124:35669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.126.keytab
--principal=kudu/127.21.143.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.126:37735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.143.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:47:00.901464 12275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:00.902451 12275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:00.910710 12280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:00.910702 12281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:00.910678 12283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:00.911154 12275 server_base.cc:1034] running on GCE node
I20260619 18:47:00.911752 12275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:00.912822 12275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:00.914007 12275 hybrid_clock.cc:648] HybridClock initialized: now 1781894820913988 us; error 36 us; skew 500 ppm
Jun 19 18:47:00 dist-test-slave-mld5 krb5kdc[12262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894820, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:00.916636 12275 init.cc:377] Logged in from keytab as kudu/127.21.143.126@KRBTEST.COM (short username kudu)
I20260619 18:47:00.917474 12275 webserver.cc:460] Webserver started at http://127.21.143.126:32863/ using document root <none> and password file <none>
I20260619 18:47:00.918032 12275 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:00.918125 12275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:00.918362 12275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:00.920393 12275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-0/data/instance:
uuid: "1e391a3c5c604f44a43089984a268c9f"
format_stamp: "Formatted at 2026-06-19 18:47:00 on dist-test-slave-mld5"
server_key: "1883d991190fff6010a1a29487050f62"
server_key_iv: "8d5199a49fb3c874460e337340a6f3c0"
server_key_version: "encryptionkey@0"
I20260619 18:47:00.920944 12275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-0/wal/instance:
uuid: "1e391a3c5c604f44a43089984a268c9f"
format_stamp: "Formatted at 2026-06-19 18:47:00 on dist-test-slave-mld5"
server_key: "1883d991190fff6010a1a29487050f62"
server_key_iv: "8d5199a49fb3c874460e337340a6f3c0"
server_key_version: "encryptionkey@0"
I20260619 18:47:00.924539 12275 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:47:00.927202 12290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:00.928383 12275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:47:00.928516 12275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-0/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-0/wal
uuid: "1e391a3c5c604f44a43089984a268c9f"
format_stamp: "Formatted at 2026-06-19 18:47:00 on dist-test-slave-mld5"
server_key: "1883d991190fff6010a1a29487050f62"
server_key_iv: "8d5199a49fb3c874460e337340a6f3c0"
server_key_version: "encryptionkey@0"
I20260619 18:47:00.928627 12275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:00.971905 12275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:00.974854 12275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:00.975073 12275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:00.981608 12275 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.126:37735
I20260619 18:47:00.981889 12342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.126:37735 every 8 connection(s)
I20260619 18:47:00.982523 12275 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-0/data/info.pb
I20260619 18:47:00.984351 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 12275
I20260619 18:47:00.984493 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-0/wal/instance
I20260619 18:47:00.984788 22077 external_mini_cluster.cc:1428] Setting key 32a9f3bb3325d54a3a8b88bead2f2548
I20260619 18:47:00.985447 12343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:00.989518 12343 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37735 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.143.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.125@KRBTEST.COM" created.
Jun 19 18:47:00 dist-test-slave-mld5 krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894820, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:47:01.004528 12346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:00.991787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40372 (local address 127.21.143.126:37735)
0619 18:47:00.992251 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:00.992270 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:00.992541 (+ 271us) server_negotiation.cc:407] Connection header received
0619 18:47:00.993568 (+ 1027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:00.993573 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:00.993758 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:00.993939 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:00.994725 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:00.995702 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:00.997162 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:00.997535 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:00.997658 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:00.999911 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:00.999917 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:00.999935 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:00.999953 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:01.001491 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.001948 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.001954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.001955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.002056 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.002356 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.002359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.002360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.002699 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:01.002834 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:01.003221 (+ 387us) server_negotiation.cc:299] Negotiation successful
0619 18:47:01.003528 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":10,"server-negotiator.queue_time_us":325,"thread_start_us":162,"threads_started":1}
I20260619 18:47:01.004525 12347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:00.991787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:37735 (local address 127.0.0.1:40372)
0619 18:47:00.992397 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:00.992431 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:00.993250 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:00.993935 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:00.993940 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:00.994356 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:00.994582 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:00.994595 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:00.995824 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:00.995830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:00.997026 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:00.997038 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:00.997676 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:00.997679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:00.997760 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:00.998360 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:00.998375 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:00.999772 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:01.001615 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:01.001625 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:01.001633 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:01.001844 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:01.002155 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:01.002158 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:01.002160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:01.002263 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:01.002817 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:01.002823 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:01.002978 (+ 155us) client_negotiation.cc:769] Sending connection context
0619 18:47:01.003150 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 18:47:01.003568 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":147,"spinlock_wait_cycles":55680,"thread_start_us":189,"threads_started":1}
I20260619 18:47:01.006868 12343 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.143.125" port: 40255 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:47:01.007608 12347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.007359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.125:40255 (local address 127.0.0.1:33638)
0619 18:47:01.007469 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:01.007546 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.125:40255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 18:47:01.007799 12292 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.143.125:40255: connect: Connection refused (error 111)
I20260619 18:47:01.008603 12347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.008383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.125:40255 (local address 127.0.0.1:33648)
0619 18:47:01.008473 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:01.008515 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.125:40255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 18:47:01.008795 12343 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.143.125:40255: Network error: Client connection negotiation failed: client connection to 127.21.143.125:40255: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.125.keytab.
Entry for principal kudu/127.21.143.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.125.keytab.
I20260619 18:47:01.009879 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.125:40255
--webserver_interface=127.21.143.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.125.keytab
--principal=kudu/127.21.143.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:39963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:37735,127.21.143.125:40255,127.21.143.124:35669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:47:01.058033 12343 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.143.125" port: 40255 } attempt: 1
I20260619 18:47:01.058921 12347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.058636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.125:40255 (local address 127.0.0.1:33664)
0619 18:47:01.058783 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:01.058847 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.125:40255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 18:47:01.059639 12347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.059436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.125:40255 (local address 127.0.0.1:33670)
0619 18:47:01.059538 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:01.059588 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.125:40255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 18:47:01.059760 12343 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.143.125:40255: Network error: Client connection negotiation failed: client connection to 127.21.143.125:40255: connect: Connection refused (error 111)
W20260619 18:47:01.120820 12350 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 18:47:01.121156 12350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:01.121230 12350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:01.121311 12350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:01.125252 12350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:47:01.125339 12350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:01.125377 12350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:01.125414 12350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:47:01.125442 12350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:47:01.127918 12343 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.143.125" port: 40255 } attempt: 2
I20260619 18:47:01.128669 12347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.128394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.125:40255 (local address 127.0.0.1:33682)
0619 18:47:01.128529 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:01.128598 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.125:40255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 18:47:01.129314 12347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.129114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.125:40255 (local address 127.0.0.1:33684)
0619 18:47:01.129217 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:01.129260 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.125:40255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 18:47:01.129433 12343 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.143.125:40255: Network error: Client connection negotiation failed: client connection to 127.21.143.125:40255: connect: Connection refused (error 111)
I20260619 18:47:01.130810 12350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:39963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:37735,127.21.143.125:40255,127.21.143.124:35669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.125.keytab
--principal=kudu/127.21.143.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.125:40255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.143.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:47:01.131863 12350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:01.132900 12350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:01.140911 12358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:01.140901 12355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:01.140946 12350 server_base.cc:1034] running on GCE node
W20260619 18:47:01.140901 12356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:01.141738 12350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:01.142743 12350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:01.143896 12350 hybrid_clock.cc:648] HybridClock initialized: now 1781894821143882 us; error 26 us; skew 500 ppm
Jun 19 18:47:01 dist-test-slave-mld5 krb5kdc[12262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894821, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:01.146644 12350 init.cc:377] Logged in from keytab as kudu/127.21.143.125@KRBTEST.COM (short username kudu)
I20260619 18:47:01.147611 12350 webserver.cc:460] Webserver started at http://127.21.143.125:42721/ using document root <none> and password file <none>
I20260619 18:47:01.148169 12350 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:01.148231 12350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:01.148443 12350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:01.150509 12350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-1/data/instance:
uuid: "7267a875f21e402bbef5ac1cda828da0"
format_stamp: "Formatted at 2026-06-19 18:47:01 on dist-test-slave-mld5"
server_key: "4bd75fb722f9e84ab9e1de18273c5668"
server_key_iv: "ab7961fae3cb7c7a2f18bd06f754ee64"
server_key_version: "encryptionkey@0"
I20260619 18:47:01.151094 12350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-1/wal/instance:
uuid: "7267a875f21e402bbef5ac1cda828da0"
format_stamp: "Formatted at 2026-06-19 18:47:01 on dist-test-slave-mld5"
server_key: "4bd75fb722f9e84ab9e1de18273c5668"
server_key_iv: "ab7961fae3cb7c7a2f18bd06f754ee64"
server_key_version: "encryptionkey@0"
I20260619 18:47:01.155089 12350 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260619 18:47:01.157684 12365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:01.158908 12350 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:47:01.159053 12350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-1/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-1/wal
uuid: "7267a875f21e402bbef5ac1cda828da0"
format_stamp: "Formatted at 2026-06-19 18:47:01 on dist-test-slave-mld5"
server_key: "4bd75fb722f9e84ab9e1de18273c5668"
server_key_iv: "ab7961fae3cb7c7a2f18bd06f754ee64"
server_key_version: "encryptionkey@0"
I20260619 18:47:01.159174 12350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:01.203922 12350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:01.207197 12350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:01.207454 12350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:01.214088 12350 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.125:40255
I20260619 18:47:01.214373 12417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.125:40255 every 8 connection(s)
I20260619 18:47:01.215097 12350 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-1/data/info.pb
I20260619 18:47:01.216063 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 12350
I20260619 18:47:01.216200 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-1/wal/instance
I20260619 18:47:01.216508 22077 external_mini_cluster.cc:1428] Setting key 61fd759d08d3c26093cbf4320d167c42
I20260619 18:47:01.217960 12418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:01.220602 12343 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.143.125" port: 40255 } attempt: 3
I20260619 18:47:01.221951 12418 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37735 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.143.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.143.124@KRBTEST.COM" created.
Jun 19 18:47:01 dist-test-slave-mld5 krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894820, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for kudu/127.21.143.125@KRBTEST.COM
Jun 19 18:47:01 dist-test-slave-mld5 krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894821, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.125@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:47:01.234812 12347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.221178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.125:40255 (local address 127.0.0.1:33696)
0619 18:47:01.221301 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:01.221319 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:01.221417 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:01.224258 (+ 2841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:01.224262 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:01.224451 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:01.224588 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:01.224599 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.226262 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.226267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:01.226959 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:01.226968 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.227822 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.227827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:01.227968 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.228555 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:01.228573 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:01.230155 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:01.232617 (+ 2462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:01.232622 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:01.232624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:01.233076 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:01.233456 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:01.233459 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:01.233460 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:01.233545 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:01.234254 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:01.234257 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:01.234368 (+ 111us) client_negotiation.cc:769] Sending connection context
0619 18:47:01.234475 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 18:47:01.234629 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 18:47:01.236330 12421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.221903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33696 (local address 127.21.143.125:40255)
0619 18:47:01.222384 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:01.222403 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:01.222450 (+ 47us) server_negotiation.cc:407] Connection header received
0619 18:47:01.223453 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:01.223478 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:01.223942 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:01.224781 (+ 839us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:01.225184 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.226148 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.227075 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.227618 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.227844 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.230290 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:01.230300 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:01.230317 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:01.230348 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:01.232508 (+ 2160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.233182 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.233188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.233195 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.233310 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.233657 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.233661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.233663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.234163 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:01.234274 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:01.234674 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 18:47:01.235413 (+ 739us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":121,"threads_started":1}
I20260619 18:47:01.236465 12346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.223681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40380 (local address 127.21.143.126:37735)
0619 18:47:01.223798 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:01.223802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:01.224148 (+ 346us) server_negotiation.cc:407] Connection header received
0619 18:47:01.224335 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:01.224339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:01.224392 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:01.224432 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:01.225457 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.226069 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.227143 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.228153 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.228253 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.230982 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:01.230985 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:01.230988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:01.231017 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:01.233261 (+ 2244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.233807 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.233810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.233814 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.233905 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.234203 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.234208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.234213 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.234445 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:01.234553 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:01.236172 (+ 1619us) server_negotiation.cc:299] Negotiation successful
0619 18:47:01.236302 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 18:47:01.236882 12422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.223668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:37735 (local address 127.0.0.1:40380)
0619 18:47:01.224008 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:01.224065 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:01.224228 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:01.224617 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:01.224623 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:01.225006 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:01.225310 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:01.225321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.226193 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.226199 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:01.227017 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:01.227032 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.228291 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.228295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:01.228446 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.228951 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:01.228972 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:01.230858 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:01.233373 (+ 2515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:01.233383 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:01.233402 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:01.233703 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:01.234010 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:01.234013 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:01.234015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:01.234101 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:01.235741 (+ 1640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:01.235751 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:01.235893 (+ 142us) client_negotiation.cc:769] Sending connection context
0619 18:47:01.236607 (+ 714us) client_negotiation.cc:240] Negotiation successful
0619 18:47:01.236754 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":80,"threads_started":1}
I20260619 18:47:01.238500 12418 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.143.125" port: 40255 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:47:01.238677 12343 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35669 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:47:01.239634 12347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.239364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:35669 (local address 127.0.0.1:35310)
0619 18:47:01.239504 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:01.239567 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 18:47:01.240336 12347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.240098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:35669 (local address 127.0.0.1:35312)
0619 18:47:01.240230 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:01.240280 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260619 18:47:01.240531 12343 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.143.124:35669: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35669: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.143.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.124.keytab.
Entry for principal kudu/127.21.143.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.124.keytab.
Jun 19 18:47:01 dist-test-slave-mld5 krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894821, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.125@KRBTEST.COM for kudu/127.21.143.125@KRBTEST.COM
I20260619 18:47:01.244321 22077 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1WQx4B/build/debug/bin/kudu
/tmp/dist-test-task1WQx4B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.124:35669
--webserver_interface=127.21.143.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.124.keytab
--principal=kudu/127.21.143.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.143.84:39963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.143.126:37735,127.21.143.125:40255,127.21.143.124:35669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:47:01.247318 12422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.239046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.125:40255 (local address 127.0.0.1:33706)
0619 18:47:01.239162 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:01.239178 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:01.239257 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:01.239548 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:01.239550 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:01.239797 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:01.239935 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:01.239943 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.240939 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.240942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:01.241644 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:01.241653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.242154 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.242157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:01.242311 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.242774 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:01.242791 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:01.244068 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:01.245842 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:01.245844 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:01.245846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:01.246033 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:01.246316 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:01.246318 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:01.246319 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:01.246388 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:01.246831 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:01.246833 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:01.246909 (+ 76us) client_negotiation.cc:769] Sending connection context
0619 18:47:01.247008 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 18:47:01.247107 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 18:47:01.247330 12421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.239197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33706 (local address 127.21.143.125:40255)
0619 18:47:01.239278 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:01.239282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:01.239327 (+ 45us) server_negotiation.cc:407] Connection header received
0619 18:47:01.239361 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:01.239364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:01.239428 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:01.239504 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:01.240067 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.240791 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.241807 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.242059 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.242137 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.244194 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:01.244197 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:01.244200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:01.244227 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:01.245727 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.246128 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.246134 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.246136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.246227 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.246478 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.246480 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.246482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.246701 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:01.246805 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:01.247068 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 18:47:01.247157 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 18:47:01.248159 12418 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35669 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:47:01.248894 12422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.248610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:35669 (local address 127.0.0.1:35316)
0619 18:47:01.248739 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:01.248825 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 18:47:01.249666 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35669: connect: Connection refused (error 111)
I20260619 18:47:01.250623 12422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.250422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:35669 (local address 127.0.0.1:35332)
0619 18:47:01.250531 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:01.250573 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 18:47:01.250867 12418 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.143.124:35669: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35669: connect: Connection refused (error 111)
I20260619 18:47:01.271723 12343 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35669 } attempt: 1
I20260619 18:47:01.272631 12347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.272294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:35669 (local address 127.0.0.1:35340)
0619 18:47:01.272460 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:01.272534 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 18:47:01.273443 12347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.273202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:35669 (local address 127.0.0.1:35356)
0619 18:47:01.273311 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:01.273363 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 18:47:01.273608 12343 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.143.124:35669: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35669: connect: Connection refused (error 111)
I20260619 18:47:01.300168 12418 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35669 } attempt: 1
I20260619 18:47:01.301096 12422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.300778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:35669 (local address 127.0.0.1:35364)
0619 18:47:01.300937 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:01.301001 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 18:47:01.301824 12422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.301600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:35669 (local address 127.0.0.1:35378)
0619 18:47:01.301713 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:01.301768 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 18:47:01.301954 12418 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.143.124:35669: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35669: connect: Connection refused (error 111)
I20260619 18:47:01.348783 12343 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35669 } attempt: 2
I20260619 18:47:01.349663 12347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.349327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:35669 (local address 127.0.0.1:35392)
0619 18:47:01.349516 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:01.349582 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 18:47:01.350548 12347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.350275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:35669 (local address 127.0.0.1:35408)
0619 18:47:01.350418 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:01.350468 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 18:47:01.350782 12343 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.143.124:35669: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35669: connect: Connection refused (error 111)
W20260619 18:47:01.358929 12424 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 18:47:01.359220 12424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:01.359318 12424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:01.359388 12424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:01.363543 12424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:47:01.363596 12424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:01.363615 12424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:01.363662 12424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:47:01.363687 12424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:47:01.369021 12424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.143.84:39963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.143.126:37735,127.21.143.125:40255,127.21.143.124:35669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/krb5kdc/kudu_127.21.143.124.keytab
--principal=kudu/127.21.143.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.143.124:35669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.143.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:47:01.370041 12424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:01.370152 12418 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35669 } attempt: 2
I20260619 18:47:01.370998 12422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.370670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:35669 (local address 127.0.0.1:35416)
0619 18:47:01.370835 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:01.370904 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 18:47:01.371284 12424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 18:47:01.371775 12422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.371538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:35669 (local address 127.0.0.1:35418)
0619 18:47:01.371651 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:01.371704 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 18:47:01.371917 12418 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.143.124:35669: Network error: Client connection negotiation failed: client connection to 127.21.143.124:35669: connect: Connection refused (error 111)
W20260619 18:47:01.380728 12433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:01.380713 12430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:01.380738 12431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:01.380940 12424 server_base.cc:1034] running on GCE node
I20260619 18:47:01.381513 12424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:01.382529 12424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:01.383679 12424 hybrid_clock.cc:648] HybridClock initialized: now 1781894821383662 us; error 27 us; skew 500 ppm
Jun 19 18:47:01 dist-test-slave-mld5 krb5kdc[12262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894821, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:01.386288 12424 init.cc:377] Logged in from keytab as kudu/127.21.143.124@KRBTEST.COM (short username kudu)
I20260619 18:47:01.387192 12424 webserver.cc:460] Webserver started at http://127.21.143.124:37757/ using document root <none> and password file <none>
I20260619 18:47:01.387780 12424 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:01.387868 12424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:01.388098 12424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:01.390105 12424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-2/data/instance:
uuid: "a4d0cd45a2b043459afa45b6d99ee2d8"
format_stamp: "Formatted at 2026-06-19 18:47:01 on dist-test-slave-mld5"
server_key: "5b23ba0f0fedc578af3791ac6589b9a5"
server_key_iv: "2729060ef1fe0f00ee4fec165fa78f60"
server_key_version: "encryptionkey@0"
I20260619 18:47:01.390646 12424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-2/wal/instance:
uuid: "a4d0cd45a2b043459afa45b6d99ee2d8"
format_stamp: "Formatted at 2026-06-19 18:47:01 on dist-test-slave-mld5"
server_key: "5b23ba0f0fedc578af3791ac6589b9a5"
server_key_iv: "2729060ef1fe0f00ee4fec165fa78f60"
server_key_version: "encryptionkey@0"
I20260619 18:47:01.394244 12424 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 18:47:01.396505 12440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:01.397658 12424 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:47:01.397778 12424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-2/data,/tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-2/wal
uuid: "a4d0cd45a2b043459afa45b6d99ee2d8"
format_stamp: "Formatted at 2026-06-19 18:47:01 on dist-test-slave-mld5"
server_key: "5b23ba0f0fedc578af3791ac6589b9a5"
server_key_iv: "2729060ef1fe0f00ee4fec165fa78f60"
server_key_version: "encryptionkey@0"
I20260619 18:47:01.397878 12424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:01.423061 12424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:01.425946 12424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:01.426149 12424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:01.433104 12424 rpc_server.cc:307] RPC server started. Bound to: 127.21.143.124:35669
I20260619 18:47:01.433382 12492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.143.124:35669 every 8 connection(s)
I20260619 18:47:01.434165 12424 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-2/data/info.pb
I20260619 18:47:01.436872 12493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:01.440580 12493 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37735 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:47:01.441642 22077 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1WQx4B/build/debug/bin/kudu as pid 12424
I20260619 18:47:01.441776 22077 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1WQx4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894542572984-22077-0/minicluster-data/master-2/wal/instance
I20260619 18:47:01.442126 22077 external_mini_cluster.cc:1428] Setting key 7109902525c7ef52851dbb864fa3938f
Jun 19 18:47:01 dist-test-slave-mld5 krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894821, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.124@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
Jun 19 18:47:01 dist-test-slave-mld5 krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.124@KRBTEST.COM
I20260619 18:47:01.449955 12343 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35669 } attempt: 3
I20260619 18:47:01.454293 12346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.442530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40384 (local address 127.21.143.126:37735)
0619 18:47:01.442695 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:01.442700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:01.443611 (+ 911us) server_negotiation.cc:407] Connection header received
0619 18:47:01.444525 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:01.444527 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:01.444575 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:01.444684 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:01.445567 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.446063 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.447148 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.447531 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.447619 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.450020 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:01.450023 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:01.450026 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:01.450084 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:01.451962 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.452624 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.452628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.452630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.452721 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.453122 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.453126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.453127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.453396 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:01.453517 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:01.454011 (+ 494us) server_negotiation.cc:299] Negotiation successful
0619 18:47:01.454128 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 18:47:01.455426 12497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.443149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35424 (local address 127.21.143.124:35669)
0619 18:47:01.443477 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:01.443496 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:01.443545 (+ 49us) server_negotiation.cc:407] Connection header received
0619 18:47:01.444278 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:01.444301 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:01.444555 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:01.444734 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:01.445466 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.446487 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.447418 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.447879 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.447988 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.450480 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:01.450485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:01.450493 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:01.450520 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:01.452528 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.453011 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.453017 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.453019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.453107 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.453382 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.453396 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.453398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.453810 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:01.454779 (+ 969us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:01.454883 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 18:47:01.455015 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":250,"spinlock_wait_cycles":2816,"thread_start_us":94,"threads_started":1}
I20260619 18:47:01.455435 12496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.442899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.126:37735 (local address 127.0.0.1:40384)
0619 18:47:01.443460 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:01.443513 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:01.444279 (+ 766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:01.444721 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:01.444726 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:01.445164 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:01.445421 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:01.445442 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.446169 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.446173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:01.447037 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:01.447046 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.447667 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.447671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:01.447821 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.448436 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:01.448451 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:01.449866 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:01.452068 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:01.452076 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:01.452085 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:01.452519 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:01.452825 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:01.452831 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:01.452834 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:01.453016 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:01.453505 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:01.453511 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:01.453783 (+ 272us) client_negotiation.cc:769] Sending connection context
0619 18:47:01.453950 (+ 167us) client_negotiation.cc:240] Negotiation successful
0619 18:47:01.454236 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":35,"thread_start_us":147,"threads_started":1}
Jun 19 18:47:01 dist-test-slave-mld5 krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894820, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.126@KRBTEST.COM for kudu/127.21.143.124@KRBTEST.COM
I20260619 18:47:01.456773 12493 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.143.125" port: 40255 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:47:01.460475 12347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.450642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:35669 (local address 127.0.0.1:35426)
0619 18:47:01.451268 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:01.451284 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:01.451411 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:01.451736 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:01.451740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:01.451958 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:01.452067 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:01.452073 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.452809 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.452812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:01.453553 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:01.453565 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.454178 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.454182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:01.454278 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.454702 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:01.454716 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:01.456214 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:01.458843 (+ 2629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:01.458847 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:01.458849 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:01.459084 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:01.459429 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:01.459432 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:01.459434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:01.459530 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:01.459942 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:01.459946 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:01.460058 (+ 112us) client_negotiation.cc:769] Sending connection context
0619 18:47:01.460184 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 18:47:01.460313 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536}
I20260619 18:47:01.460493 12498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.450453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35426 (local address 127.21.143.124:35669)
0619 18:47:01.450791 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:01.450799 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:01.451373 (+ 574us) server_negotiation.cc:407] Connection header received
0619 18:47:01.451548 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:01.451551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:01.451604 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:01.451703 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:01.452153 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.452680 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.453694 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.454049 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.454470 (+ 421us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.456344 (+ 1874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:01.456348 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:01.456350 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:01.456373 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:01.458746 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.459179 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.459183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.459185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.459294 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.459633 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.459636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.459637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.459838 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:01.459927 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:01.460235 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 18:47:01.460344 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":118,"threads_started":1}
Jun 19 18:47:01 dist-test-slave-mld5 krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894821, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.124@KRBTEST.COM for kudu/127.21.143.125@KRBTEST.COM
I20260619 18:47:01.463079 12418 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35669 } attempt: 3
Jun 19 18:47:01 dist-test-slave-mld5 krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.126@KRBTEST.COM
I20260619 18:47:01.463937 12343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e391a3c5c604f44a43089984a268c9f: Bootstrap starting.
I20260619 18:47:01.466784 12343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e391a3c5c604f44a43089984a268c9f: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:01.467710 12343 log.cc:824] T 00000000000000000000000000000000 P 1e391a3c5c604f44a43089984a268c9f: Log is configured to *not* fsync() on all Append() calls
I20260619 18:47:01.467914 12421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.457340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33708 (local address 127.21.143.125:40255)
0619 18:47:01.457459 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:01.457464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:01.457673 (+ 209us) server_negotiation.cc:407] Connection header received
0619 18:47:01.457821 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:01.457826 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:01.457882 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:01.457937 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:01.458538 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.459259 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.460067 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.460419 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.460524 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.462989 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:01.462992 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:01.462994 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:01.463020 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:01.464608 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.465407 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.465413 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.465681 (+ 268us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.465777 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.466127 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.466130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.466134 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.467200 (+ 1066us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:01.467288 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:01.467613 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 18:47:01.467740 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":850,"server-negotiator.queue_time_us":47}
I20260619 18:47:01.468578 12346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.457885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40386 (local address 127.21.143.126:37735)
0619 18:47:01.457981 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:01.457984 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:01.458115 (+ 131us) server_negotiation.cc:407] Connection header received
0619 18:47:01.458216 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:01.458219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:01.458264 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:01.458341 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:01.459034 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.459736 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.460788 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.461156 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.461266 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.464591 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:01.464595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:01.464597 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:01.464621 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:01.466361 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.466756 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.466759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.466761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.466866 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.467165 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.467169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.467171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.467417 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:01.468208 (+ 791us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:01.468290 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 18:47:01.468412 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 18:47:01.468976 12496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.457366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.125:40255 (local address 127.0.0.1:33708)
0619 18:47:01.457583 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:01.457596 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:01.457684 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:01.458111 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:01.458113 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:01.458328 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:01.458418 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:01.458424 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.459406 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.459410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:01.459948 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:01.459957 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.460757 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.460760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:01.460843 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.461278 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:01.461293 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:01.462843 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:01.464713 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:01.464716 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:01.464718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:01.465281 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:01.465911 (+ 630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:01.465916 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:01.465917 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:01.466013 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:01.467327 (+ 1314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:01.467331 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:01.467414 (+ 83us) client_negotiation.cc:769] Sending connection context
0619 18:47:01.468522 (+ 1108us) client_negotiation.cc:240] Negotiation successful
0619 18:47:01.468838 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
Jun 19 18:47:01 dist-test-slave-mld5 krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894821, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.125@KRBTEST.COM for kudu/127.21.143.124@KRBTEST.COM
I20260619 18:47:01.469897 12343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e391a3c5c604f44a43089984a268c9f: No bootstrap required, opened a new log
I20260619 18:47:01.470042 12493 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35669 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:47:01.473318 12422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.463751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:35669 (local address 127.0.0.1:35430)
0619 18:47:01.463855 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:01.463869 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:01.463937 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:01.464938 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:01.464940 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:01.465101 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:01.465205 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:01.465213 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.466213 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.466216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:01.466981 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:01.466994 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.467887 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.467891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:01.467967 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.468393 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:01.468410 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:01.469762 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:01.471863 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:01.471866 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:01.471868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:01.472094 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:01.472393 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:01.472396 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:01.472397 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:01.472490 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:01.472883 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:01.472886 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:01.472969 (+ 83us) client_negotiation.cc:769] Sending connection context
0619 18:47:01.473046 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 18:47:01.473164 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":3}
I20260619 18:47:01.473910 12498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.463863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35430 (local address 127.21.143.124:35669)
0619 18:47:01.464744 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:01.464748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:01.464759 (+ 11us) server_negotiation.cc:407] Connection header received
0619 18:47:01.464790 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:01.464793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:01.464829 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:01.465320 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:01.465408 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.466086 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.467383 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.467760 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.468159 (+ 399us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.469901 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:01.469905 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:01.469911 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:01.469939 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:01.471740 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.472213 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.472218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.472219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.472294 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.472597 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.472600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.472602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.472779 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:01.473589 (+ 810us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:01.473665 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 18:47:01.473769 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 18:47:01.474431 12343 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e391a3c5c604f44a43089984a268c9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e391a3c5c604f44a43089984a268c9f" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37735 } } peers { permanent_uuid: "7267a875f21e402bbef5ac1cda828da0" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 40255 } } peers { permanent_uuid: "a4d0cd45a2b043459afa45b6d99ee2d8" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35669 } }
I20260619 18:47:01.474653 12343 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e391a3c5c604f44a43089984a268c9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e391a3c5c604f44a43089984a268c9f, State: Initialized, Role: FOLLOWER
I20260619 18:47:01.474982 12343 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e391a3c5c604f44a43089984a268c9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e391a3c5c604f44a43089984a268c9f" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37735 } } peers { permanent_uuid: "7267a875f21e402bbef5ac1cda828da0" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 40255 } } peers { permanent_uuid: "a4d0cd45a2b043459afa45b6d99ee2d8" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35669 } }
Jun 19 18:47:01 dist-test-slave-mld5 krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.143.125@KRBTEST.COM
Jun 19 18:47:01 dist-test-slave-mld5 krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894821, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.143.124@KRBTEST.COM for kudu/127.21.143.124@KRBTEST.COM
I20260619 18:47:01.476343 12343 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e391a3c5c604f44a43089984a268c9f [sys.catalog]: configured and running, proceeding with master startup.
W20260619 18:47:01.476588 12296 tablet.cc:2369] T 00000000000000000000000000000000 P 1e391a3c5c604f44a43089984a268c9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:47:01.476824 12418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7267a875f21e402bbef5ac1cda828da0: Bootstrap starting.
I20260619 18:47:01.479736 12418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7267a875f21e402bbef5ac1cda828da0: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:01.480590 12418 log.cc:824] T 00000000000000000000000000000000 P 7267a875f21e402bbef5ac1cda828da0: Log is configured to *not* fsync() on all Append() calls
I20260619 18:47:01.481106 12496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.470454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.143.124:35669 (local address 127.0.0.1:35440)
0619 18:47:01.470572 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:01.470584 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:01.470677 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:01.470962 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:01.470966 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:01.471189 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:01.471286 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:01.471292 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.472317 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.472322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:01.473110 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:01.473120 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.474223 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.474231 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:01.474339 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.474877 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:01.474895 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:01.477202 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:01.479581 (+ 2379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:01.479586 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:01.479588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:01.479823 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:01.480171 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:01.480173 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:01.480176 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:01.480245 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:01.480646 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:01.480648 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:01.480721 (+ 73us) client_negotiation.cc:769] Sending connection context
0619 18:47:01.480822 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 18:47:01.480938 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 18:47:01.481312 12421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.469777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33724 (local address 127.21.143.125:40255)
0619 18:47:01.469898 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:01.469901 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:01.470156 (+ 255us) server_negotiation.cc:407] Connection header received
0619 18:47:01.470574 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:01.470579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:01.470629 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:01.470681 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:01.471562 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.472247 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.473260 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.473631 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.473705 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.477367 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:01.477370 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:01.477373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:01.477396 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:01.479116 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.479632 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.479637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.479639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.479713 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.480010 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.480013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.480015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.480218 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:01.480411 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:01.481052 (+ 641us) server_negotiation.cc:299] Negotiation successful
0619 18:47:01.481151 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 18:47:01.481875 12497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.470478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35440 (local address 127.21.143.124:35669)
0619 18:47:01.470567 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:01.470571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:01.470629 (+ 58us) server_negotiation.cc:407] Connection header received
0619 18:47:01.470761 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:01.470767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:01.470802 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:01.470844 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:01.471434 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.472196 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.473236 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.473813 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.473878 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.477327 (+ 3449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:01.477333 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:01.477336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:01.477358 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:01.479472 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.479981 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.479985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.479989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.480080 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.480335 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.480338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.480339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.480521 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:01.480609 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:01.481616 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0619 18:47:01.481720 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 18:47:01.482949 12418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7267a875f21e402bbef5ac1cda828da0: No bootstrap required, opened a new log
I20260619 18:47:01.484711 12493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4d0cd45a2b043459afa45b6d99ee2d8: Bootstrap starting.
I20260619 18:47:01.481128 12275 master_runner.cc:186] Error getting master registration for 127.21.143.125:40255: OK, instance_id { permanent_uuid: "7267a875f21e402bbef5ac1cda828da0" instance_seqno: 1781894821212239 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 18:47:01.484864 12275 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 18:47:01.485122 12500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e391a3c5c604f44a43089984a268c9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e391a3c5c604f44a43089984a268c9f" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37735 } } peers { permanent_uuid: "7267a875f21e402bbef5ac1cda828da0" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 40255 } } peers { permanent_uuid: "a4d0cd45a2b043459afa45b6d99ee2d8" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35669 } } }
I20260619 18:47:01.485235 12500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e391a3c5c604f44a43089984a268c9f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:47:01.486522 12418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7267a875f21e402bbef5ac1cda828da0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e391a3c5c604f44a43089984a268c9f" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37735 } } peers { permanent_uuid: "7267a875f21e402bbef5ac1cda828da0" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 40255 } } peers { permanent_uuid: "a4d0cd45a2b043459afa45b6d99ee2d8" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35669 } }
I20260619 18:47:01.486685 12418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7267a875f21e402bbef5ac1cda828da0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7267a875f21e402bbef5ac1cda828da0, State: Initialized, Role: FOLLOWER
I20260619 18:47:01.486874 12493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4d0cd45a2b043459afa45b6d99ee2d8: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:01.487011 12418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7267a875f21e402bbef5ac1cda828da0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e391a3c5c604f44a43089984a268c9f" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37735 } } peers { permanent_uuid: "7267a875f21e402bbef5ac1cda828da0" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 40255 } } peers { permanent_uuid: "a4d0cd45a2b043459afa45b6d99ee2d8" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35669 } }
I20260619 18:47:01.487509 12513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7267a875f21e402bbef5ac1cda828da0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e391a3c5c604f44a43089984a268c9f" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37735 } } peers { permanent_uuid: "7267a875f21e402bbef5ac1cda828da0" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 40255 } } peers { permanent_uuid: "a4d0cd45a2b043459afa45b6d99ee2d8" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35669 } } }
I20260619 18:47:01.487634 12513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7267a875f21e402bbef5ac1cda828da0 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:47:01.487779 12493 log.cc:824] T 00000000000000000000000000000000 P a4d0cd45a2b043459afa45b6d99ee2d8: Log is configured to *not* fsync() on all Append() calls
I20260619 18:47:01.487825 22077 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:47:01.488371 12418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7267a875f21e402bbef5ac1cda828da0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:47:01.489041 12317 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1e391a3c5c604f44a43089984a268c9f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40386
I20260619 18:47:01.489284 12317 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e391a3c5c604f44a43089984a268c9f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 18:47:01.489411 12317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e391a3c5c604f44a43089984a268c9f [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:01.489773 12493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4d0cd45a2b043459afa45b6d99ee2d8: No bootstrap required, opened a new log
I20260619 18:47:01.490823 12317 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e391a3c5c604f44a43089984a268c9f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e391a3c5c604f44a43089984a268c9f" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37735 } } peers { permanent_uuid: "7267a875f21e402bbef5ac1cda828da0" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 40255 } } peers { permanent_uuid: "a4d0cd45a2b043459afa45b6d99ee2d8" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35669 } }
W20260619 18:47:01.491050 12511 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1e391a3c5c604f44a43089984a268c9f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:47:01.491119 12511 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:47:01.491921 12317 leader_election.cc:288] T 00000000000000000000000000000000 P 1e391a3c5c604f44a43089984a268c9f [CANDIDATE]: Term 1 election: Requested vote from peers 7267a875f21e402bbef5ac1cda828da0 (127.21.143.125:40255), a4d0cd45a2b043459afa45b6d99ee2d8 (127.21.143.124:35669)
I20260619 18:47:01.493436 12392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e391a3c5c604f44a43089984a268c9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7267a875f21e402bbef5ac1cda828da0"
I20260619 18:47:01.493669 12392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7267a875f21e402bbef5ac1cda828da0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:01.494032 12493 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4d0cd45a2b043459afa45b6d99ee2d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e391a3c5c604f44a43089984a268c9f" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37735 } } peers { permanent_uuid: "7267a875f21e402bbef5ac1cda828da0" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 40255 } } peers { permanent_uuid: "a4d0cd45a2b043459afa45b6d99ee2d8" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35669 } }
I20260619 18:47:01.494270 12493 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4d0cd45a2b043459afa45b6d99ee2d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4d0cd45a2b043459afa45b6d99ee2d8, State: Initialized, Role: FOLLOWER
I20260619 18:47:01.494608 12493 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4d0cd45a2b043459afa45b6d99ee2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e391a3c5c604f44a43089984a268c9f" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37735 } } peers { permanent_uuid: "7267a875f21e402bbef5ac1cda828da0" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 40255 } } peers { permanent_uuid: "a4d0cd45a2b043459afa45b6d99ee2d8" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35669 } }
W20260619 18:47:01.495244 12526 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7267a875f21e402bbef5ac1cda828da0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:47:01.495373 12526 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:47:01.495630 12392 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7267a875f21e402bbef5ac1cda828da0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e391a3c5c604f44a43089984a268c9f in term 1.
I20260619 18:47:01.496080 12493 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4d0cd45a2b043459afa45b6d99ee2d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:47:01.496165 12467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e391a3c5c604f44a43089984a268c9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a4d0cd45a2b043459afa45b6d99ee2d8"
I20260619 18:47:01.496400 12467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4d0cd45a2b043459afa45b6d99ee2d8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:01.497099 12292 leader_election.cc:302] T 00000000000000000000000000000000 P 1e391a3c5c604f44a43089984a268c9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e391a3c5c604f44a43089984a268c9f, 7267a875f21e402bbef5ac1cda828da0; no voters:
I20260619 18:47:01.497399 12500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e391a3c5c604f44a43089984a268c9f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:47:01.497605 12500 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e391a3c5c604f44a43089984a268c9f [term 1 LEADER]: Becoming Leader. State: Replica: 1e391a3c5c604f44a43089984a268c9f, State: Running, Role: LEADER
I20260619 18:47:01.497923 12500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e391a3c5c604f44a43089984a268c9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e391a3c5c604f44a43089984a268c9f" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37735 } } peers { permanent_uuid: "7267a875f21e402bbef5ac1cda828da0" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 40255 } } peers { permanent_uuid: "a4d0cd45a2b043459afa45b6d99ee2d8" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35669 } }
I20260619 18:47:01.498231 12467 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a4d0cd45a2b043459afa45b6d99ee2d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e391a3c5c604f44a43089984a268c9f in term 1.
I20260619 18:47:01.499827 12500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e391a3c5c604f44a43089984a268c9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e391a3c5c604f44a43089984a268c9f. Latest consensus state: current_term: 1 leader_uuid: "1e391a3c5c604f44a43089984a268c9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e391a3c5c604f44a43089984a268c9f" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37735 } } peers { permanent_uuid: "7267a875f21e402bbef5ac1cda828da0" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 40255 } } peers { permanent_uuid: "a4d0cd45a2b043459afa45b6d99ee2d8" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35669 } } }
I20260619 18:47:01.499981 12500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e391a3c5c604f44a43089984a268c9f [sys.catalog]: This master's current role is: LEADER
W20260619 18:47:01.502341 12547 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a4d0cd45a2b043459afa45b6d99ee2d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:47:01.502420 12547 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:47:01.505084 12549 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:47:01.505116 12350 master_runner.cc:263] No leader master found from master 7267a875f21e402bbef5ac1cda828da0
I20260619 18:47:01.505182 12350 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 18:47:01.506567 12549 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:47:01.507913 12532 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d0cd45a2b043459afa45b6d99ee2d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e391a3c5c604f44a43089984a268c9f" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37735 } } peers { permanent_uuid: "7267a875f21e402bbef5ac1cda828da0" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 40255 } } peers { permanent_uuid: "a4d0cd45a2b043459afa45b6d99ee2d8" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35669 } } }
I20260619 18:47:01.508069 12532 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d0cd45a2b043459afa45b6d99ee2d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:47:01.511181 12346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.495092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40396 (local address 127.21.143.126:37735)
0619 18:47:01.495203 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:01.495207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:01.495983 (+ 776us) server_negotiation.cc:407] Connection header received
0619 18:47:01.498852 (+ 2869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:01.498856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:01.498906 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:01.498955 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:01.500108 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.500945 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.502130 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.502442 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.502535 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.504858 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:01.504863 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:01.504865 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:01.504893 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:01.509270 (+ 4377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.509797 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.509801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.509803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.509891 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.510189 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.510192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.510194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.510412 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:01.510544 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:01.510900 (+ 356us) server_negotiation.cc:299] Negotiation successful
0619 18:47:01.511027 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 18:47:01.515702 12467 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a4d0cd45a2b043459afa45b6d99ee2d8 [term 1 FOLLOWER]: Refusing update from remote peer 1e391a3c5c604f44a43089984a268c9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:47:01.515733 12392 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7267a875f21e402bbef5ac1cda828da0 [term 1 FOLLOWER]: Refusing update from remote peer 1e391a3c5c604f44a43089984a268c9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:47:01.516374 12541 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1e391a3c5c604f44a43089984a268c9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7267a875f21e402bbef5ac1cda828da0" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 40255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:47:01.516578 12500 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1e391a3c5c604f44a43089984a268c9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4d0cd45a2b043459afa45b6d99ee2d8" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:47:01.517547 12421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.496414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33728 (local address 127.21.143.125:40255)
0619 18:47:01.496531 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:01.496535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:01.507611 (+ 11076us) server_negotiation.cc:407] Connection header received
0619 18:47:01.507763 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:01.507768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:01.507820 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:01.507892 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:01.509583 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.510357 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.511578 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.511914 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.511987 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.513704 (+ 1717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:01.513707 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:01.513710 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:01.513736 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:01.515413 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.516046 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.516049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.516051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.516140 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.516446 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.516449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.516451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.516673 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:01.516787 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:01.517263 (+ 476us) server_negotiation.cc:299] Negotiation successful
0619 18:47:01.517372 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 18:47:01.520231 12513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7267a875f21e402bbef5ac1cda828da0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e391a3c5c604f44a43089984a268c9f. Latest consensus state: current_term: 1 leader_uuid: "1e391a3c5c604f44a43089984a268c9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e391a3c5c604f44a43089984a268c9f" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37735 } } peers { permanent_uuid: "7267a875f21e402bbef5ac1cda828da0" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 40255 } } peers { permanent_uuid: "a4d0cd45a2b043459afa45b6d99ee2d8" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35669 } } }
I20260619 18:47:01.520354 12513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7267a875f21e402bbef5ac1cda828da0 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:47:01.521420 12532 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d0cd45a2b043459afa45b6d99ee2d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e391a3c5c604f44a43089984a268c9f. Latest consensus state: current_term: 1 leader_uuid: "1e391a3c5c604f44a43089984a268c9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e391a3c5c604f44a43089984a268c9f" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37735 } } peers { permanent_uuid: "7267a875f21e402bbef5ac1cda828da0" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 40255 } } peers { permanent_uuid: "a4d0cd45a2b043459afa45b6d99ee2d8" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35669 } } }
I20260619 18:47:01.521548 12532 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d0cd45a2b043459afa45b6d99ee2d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:47:01.522637 12500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e391a3c5c604f44a43089984a268c9f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1e391a3c5c604f44a43089984a268c9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e391a3c5c604f44a43089984a268c9f" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37735 } } peers { permanent_uuid: "7267a875f21e402bbef5ac1cda828da0" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 40255 } } peers { permanent_uuid: "a4d0cd45a2b043459afa45b6d99ee2d8" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35669 } } }
I20260619 18:47:01.522760 12500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e391a3c5c604f44a43089984a268c9f [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:01.523574 12513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7267a875f21e402bbef5ac1cda828da0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1e391a3c5c604f44a43089984a268c9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e391a3c5c604f44a43089984a268c9f" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37735 } } peers { permanent_uuid: "7267a875f21e402bbef5ac1cda828da0" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 40255 } } peers { permanent_uuid: "a4d0cd45a2b043459afa45b6d99ee2d8" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35669 } } }
I20260619 18:47:01.523686 12513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7267a875f21e402bbef5ac1cda828da0 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:47:01.523881 12551 mvcc.cc:205] Tried to move back new op lower bound from 7298641188921663488 to 7298641188858851328. Current Snapshot: MvccSnapshot[applied={T|T < 7298641188921663488}]
I20260619 18:47:01.526098 12553 mvcc.cc:205] Tried to move back new op lower bound from 7298641188921663488 to 7298641188858851328. Current Snapshot: MvccSnapshot[applied={T|T < 7298641188921663488}]
I20260619 18:47:01.526684 12497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.496389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35446 (local address 127.21.143.124:35669)
0619 18:47:01.496503 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:01.496508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:01.511996 (+ 15488us) server_negotiation.cc:407] Connection header received
0619 18:47:01.512177 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:01.512180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:01.512228 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:01.512279 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:01.518947 (+ 6668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.519681 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.520812 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.521112 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.521186 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.522924 (+ 1738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:01.522928 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:01.522931 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:01.522957 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:01.524765 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.525254 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.525257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.525259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.525339 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.525663 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.525666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.525668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.525870 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:01.525970 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:01.526398 (+ 428us) server_negotiation.cc:299] Negotiation successful
0619 18:47:01.526524 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 18:47:01.528311 12549 catalog_manager.cc:1347] Generated new cluster ID: f30adb7909dc4e2cb259f07b54d86a78
I20260619 18:47:01.528381 12549 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:47:01.536556 12541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e391a3c5c604f44a43089984a268c9f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1e391a3c5c604f44a43089984a268c9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e391a3c5c604f44a43089984a268c9f" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37735 } } peers { permanent_uuid: "7267a875f21e402bbef5ac1cda828da0" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 40255 } } peers { permanent_uuid: "a4d0cd45a2b043459afa45b6d99ee2d8" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35669 } } }
I20260619 18:47:01.536696 12541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e391a3c5c604f44a43089984a268c9f [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:01.538197 12532 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d0cd45a2b043459afa45b6d99ee2d8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1e391a3c5c604f44a43089984a268c9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e391a3c5c604f44a43089984a268c9f" member_type: VOTER last_known_addr { host: "127.21.143.126" port: 37735 } } peers { permanent_uuid: "7267a875f21e402bbef5ac1cda828da0" member_type: VOTER last_known_addr { host: "127.21.143.125" port: 40255 } } peers { permanent_uuid: "a4d0cd45a2b043459afa45b6d99ee2d8" member_type: VOTER last_known_addr { host: "127.21.143.124" port: 35669 } } }
I20260619 18:47:01.538331 12532 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d0cd45a2b043459afa45b6d99ee2d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:47:01.557817 12549 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:47:01.559149 12549 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:47:01.584378 12549 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e391a3c5c604f44a43089984a268c9f: Generated new TSK 0
I20260619 18:47:01.585273 12549 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 18:47:01.622391 12346 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:47:01.619064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40404 (local address 127.21.143.126:37735)
0619 18:47:01.619433 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:01.619441 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:01.621359 (+ 1918us) server_negotiation.cc:407] Connection header received
0619 18:47:01.621511 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:01.621515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:01.621573 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:01.621654 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:01.622252 (+ 598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40404: BlockingRecv error: recv got EOF from 127.0.0.1:40404 (error 108)
Metrics: {"server-negotiator.queue_time_us":259}
W20260619 18:47:01.623782 12497 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:47:01.620837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35460 (local address 127.21.143.124:35669)
0619 18:47:01.620947 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:01.620953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:01.622854 (+ 1901us) server_negotiation.cc:407] Connection header received
0619 18:47:01.623033 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:01.623036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:01.623087 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:01.623137 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:01.623650 (+ 513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35460: BlockingRecv error: recv got EOF from 127.0.0.1:35460 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260619 18:47:01.631920 12421 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:47:01.624545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33736 (local address 127.21.143.125:40255)
0619 18:47:01.624663 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:01.624667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:01.624681 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:47:01.624723 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:01.624726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:01.624770 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:01.624852 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:01.631787 (+ 6935us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33736: BlockingRecv error: recv got EOF from 127.0.0.1:33736 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260619 18:47:01.656776 12421 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:47:01.639722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33752 (local address 127.21.143.125:40255)
0619 18:47:01.639842 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:01.639847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:01.655652 (+ 15805us) server_negotiation.cc:407] Connection header received
0619 18:47:01.655923 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:01.655928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:01.656020 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:01.656080 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:01.656654 (+ 574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33752: BlockingRecv error: recv got EOF from 127.0.0.1:33752 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260619 18:47:01.673918 12421 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:47:01.672143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33760 (local address 127.21.143.125:40255)
0619 18:47:01.672272 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:01.672277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:01.672885 (+ 608us) server_negotiation.cc:407] Connection header received
0619 18:47:01.673114 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:01.673118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:01.673176 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:01.673229 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:01.673796 (+ 567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33760: BlockingRecv error: recv got EOF from 127.0.0.1:33760 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 18:47:01.689482 12421 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:47:01.683679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33764 (local address 127.21.143.125:40255)
0619 18:47:01.683798 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:01.683802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:01.688387 (+ 4585us) server_negotiation.cc:407] Connection header received
0619 18:47:01.688604 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:01.688607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:01.688666 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:01.688719 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:01.689361 (+ 642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33764: BlockingRecv error: recv got EOF from 127.0.0.1:33764 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260619 18:47:01.705555 12421 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:47:01.703755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33766 (local address 127.21.143.125:40255)
0619 18:47:01.703924 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:01.703929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:01.704517 (+ 588us) server_negotiation.cc:407] Connection header received
0619 18:47:01.704739 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:01.704743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:01.704803 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:01.704863 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:01.705429 (+ 566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33766: BlockingRecv error: recv got EOF from 127.0.0.1:33766 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260619 18:47:01.728996 12421 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:47:01.715947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33770 (local address 127.21.143.125:40255)
0619 18:47:01.716081 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:01.716086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:01.728028 (+ 11942us) server_negotiation.cc:407] Connection header received
0619 18:47:01.728241 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:01.728244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:01.728299 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:01.728350 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:01.728877 (+ 527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33770: BlockingRecv error: recv got EOF from 127.0.0.1:33770 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 18:47:01.750932 12421 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:47:01.747955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33784 (local address 127.21.143.125:40255)
0619 18:47:01.748081 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:01.748086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:01.750061 (+ 1975us) server_negotiation.cc:407] Connection header received
0619 18:47:01.750224 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:01.750227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:01.750282 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:01.750376 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:01.750811 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33784: BlockingRecv error: recv got EOF from 127.0.0.1:33784 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 18:47:01.786754 12421 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:47:01.785416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33794 (local address 127.21.143.125:40255)
0619 18:47:01.785579 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:01.785585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:01.785784 (+ 199us) server_negotiation.cc:407] Connection header received
0619 18:47:01.785990 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:01.785993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:01.786048 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:01.786140 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:01.786611 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33794: BlockingRecv error: recv got EOF from 127.0.0.1:33794 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 18:47:01.854104 12421 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:47:01.852882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33800 (local address 127.21.143.125:40255)
0619 18:47:01.853067 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:01.853073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:01.853151 (+ 78us) server_negotiation.cc:407] Connection header received
0619 18:47:01.853258 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:01.853264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:01.853342 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:01.853483 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:01.853945 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33800: BlockingRecv error: recv got EOF from 127.0.0.1:33800 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260619 18:47:01.985116 12421 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:47:01.983938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33814 (local address 127.21.143.125:40255)
0619 18:47:01.984083 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:01.984090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:01.984258 (+ 168us) server_negotiation.cc:407] Connection header received
0619 18:47:01.984393 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:01.984395 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:01.984443 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:01.984511 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:01.984984 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33814: BlockingRecv error: recv got EOF from 127.0.0.1:33814 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 18:47:02.244272 12421 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:47:02.243396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33822 (local address 127.21.143.125:40255)
0619 18:47:02.243534 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:02.243537 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:02.243552 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:47:02.243631 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:02.243634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:02.243681 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:02.243761 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:02.244178 (+ 417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33822: BlockingRecv error: recv got EOF from 127.0.0.1:33822 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 18:47:02.496762 12526 catalog_manager.cc:1259] Loaded cluster ID: f30adb7909dc4e2cb259f07b54d86a78
I20260619 18:47:02.496840 12526 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7267a875f21e402bbef5ac1cda828da0: loading cluster ID for follower catalog manager: success
I20260619 18:47:02.498570 12526 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7267a875f21e402bbef5ac1cda828da0: acquiring CA information for follower catalog manager: success
I20260619 18:47:02.499817 12526 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7267a875f21e402bbef5ac1cda828da0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 18:47:02.503435 12547 catalog_manager.cc:1259] Loaded cluster ID: f30adb7909dc4e2cb259f07b54d86a78
I20260619 18:47:02.503487 12547 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a4d0cd45a2b043459afa45b6d99ee2d8: loading cluster ID for follower catalog manager: success
I20260619 18:47:02.505033 12547 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a4d0cd45a2b043459afa45b6d99ee2d8: acquiring CA information for follower catalog manager: success
I20260619 18:47:02.506098 12547 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a4d0cd45a2b043459afa45b6d99ee2d8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 18:47:02.762049 12623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:02.758290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33828 (local address 127.21.143.125:40255)
0619 18:47:02.758518 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:02.758521 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:02.758577 (+ 56us) server_negotiation.cc:407] Connection header received
0619 18:47:02.758711 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:02.758714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:02.758834 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:02.758930 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:47:02.759142 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.759682 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.760647 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.760871 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.760949 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:02.761207 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:47:02.761650 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:47:02.761739 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:02.761805 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 18:47:02.761852 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":60,"threads_started":1}
I20260619 18:47:02.768361 12629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:02.765000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35464 (local address 127.21.143.124:35669)
0619 18:47:02.765209 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:02.765212 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:02.765224 (+ 12us) server_negotiation.cc:407] Connection header received
0619 18:47:02.765299 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:02.765300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:02.765407 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:02.765502 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:47:02.765689 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.766184 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.766969 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.767181 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.767254 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:02.767565 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:47:02.767974 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:47:02.768068 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:02.768120 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 18:47:02.768166 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":68,"threads_started":1}
I20260619 18:47:02.775743 12629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:02.771904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35470 (local address 127.21.143.124:35669)
0619 18:47:02.772013 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:02.772017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:02.772031 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:47:02.772078 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:02.772079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:02.772113 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:02.772192 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:47:02.772496 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.773269 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.774458 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.774784 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.774881 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:02.775155 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:47:02.775235 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:47:02.775328 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:02.775428 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 18:47:02.775475 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 18:47:02.776252 12637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:02.771662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40414 (local address 127.21.143.126:37735)
0619 18:47:02.772036 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:02.772041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:02.772055 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:47:02.772153 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:02.772156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:02.772333 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:02.772461 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:47:02.772896 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.773639 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.774516 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.774860 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.774946 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:02.775388 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:47:02.775755 (+ 367us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:47:02.775843 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:02.775995 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 18:47:02.776104 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":104,"threads_started":1}
I20260619 18:47:02.776340 12623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:02.771241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33840 (local address 127.21.143.125:40255)
0619 18:47:02.771398 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:02.771403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:02.771612 (+ 209us) server_negotiation.cc:407] Connection header received
0619 18:47:02.771799 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:02.771802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:02.771862 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:02.771942 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:47:02.773266 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.774079 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.775022 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.775356 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.775447 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:02.775767 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:47:02.775850 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:47:02.775928 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:02.776119 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 18:47:02.776184 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 18:47:02.779474 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 12275
I20260619 18:47:02.785840 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 12350
I20260619 18:47:02.791859 22077 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1WQx4B/build/debug/bin/kudu with pid 12424
2026-06-19T18:47:02Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3180 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6269 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (280213 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST